rgw: complete versioning enablement after sending it to meta master

[oritwas]

Fixes: http://tracker.ceph.com/issues/18003
Signed-off-by: Orit Wasserman owasserm@redhat.com

[cbodley]

added a test case for this to test_multi.py and verified this fix 👍

there is some potential for races between the secondary zone's call to put_bucket_instance_info() and its sync from the master

consider a racing put_acls operation on the same bucket instance that happens on the master zone, which results in master mdlog entries [set_bucket_versioning, put_acls]. if the secondary zone processes the meta sync for both of these entries before calling put_bucket_instance_info(), it would overwrite the put_acls

on the other hand, say the put_acls operation happens first and the master mdlog looked like [put_acls, set_bucket_versioning]. if the secondary zone calls put_bucket_instance_info() first, then the metadata synced from the master would appear to go back in time for the put_acls until it caught back up to the set_bucket_versioning

addressing these races is probably not in scope for this bug fix, but we should think about how to handle it for all of the bucket requests that we forward

[idealguo]

I have one question: if meta_master returns success and local zone(zone group) is failed for some reason. RGW will return failed to user, but the versioning state has been updated in meta_master, it is inconsistency for user.

[oritwas]

It should be a temporary situation until the sync process run than the state will be consist again

[idealguo]

Thanks, if user got failed, and then he can find the new versioning state by API after sync process run. This may be unreasonable

[oritwas]

teuthology run: http://pulpito.ceph.com/owasserm-2016-12-26_12:31:13-rgw-wip-rgw-fix-secondary_versioning---basic-smithi/ didn't have a new issues

[oritwas]

@yehudasa , can we merge this?