HDDS-5839. Make sure buckets created from OFS are in FILE_SYSTEM_OPTI…

[JyotinderSingh]

…MIZED layout

What changes were proposed in this pull request?

The idea of this task is to change the default layout behavior of ofs client, it should by default create bucket in FILE_SYSTEM_OPTIMIZED layout.

Presently, client is sending BucketLayout.DEFAULT, which is LEGACY to the OM. This has to be modified to FILE_SYSTEM_OPTIMIZED.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5839

How was this patch tested?

Related integration tests

[bharatviswa504]

LGTM.
I have one comment posted inline.

[rakeshadr]

Good Work @JyotinderSingh. Added few comments, please go through it. Thanks!

[JyotinderSingh]

I've updated the patch with the requested changes @rakeshadr @bharatviswa504
Currently on my local I can see partial test failures in TestOzoneFileInterfaces.java, TestOzoneFSWithObjectStoreCreate.java, TestRootedOzoneFileSystem.java, TestOzoneFileSystem.java. These seem to mainly be due to some behaviour changes introduced by the patch.
The MR Acceptance test also seems to be failing due to similar reasons, I believe some updates to it would also be required from my side.

[aryangupta1998]

Thanks for the patch @JyotinderSingh. I have left few minor comments.

[rakeshadr]

I'm adding an upgrade case for wider discussion:

Assume, User OM has upgraded to OM_1.2.0 with ozone.default.bucket.layout=OBS. Now, OM will set default bucket layout to OBS if a client creates a new bucket without specifying bucket layout argument.

case-1)
step-1) An old_client_1.1.0 creates a new bucket BUCKET_01 in OM_1.2.0 and the bucket will be created successfully with default bucket layout OBS.

step-2) Now, new_client_1.2.0 performs o3fs/ofs operation on this new BUCKET_01 then it will throw validation error.

During upgrade there could be chances of both old_client_1.1.0 and new_client_1.2.0 co-exists, should we change the default value to ozone.default.bucket.layout=LEGACY ? With this the o3fs/ofs client will be happy and avoids validation errors.

This is the reason(ozone.default.bucket.layout=OBS) for many test failures in our CI.

@bharatviswa504, @mukul1987 welcome thoughts, thanks!

[bharatviswa504]

I'm adding an upgrade case for wider discussion:

Assume, User OM has upgraded to OM_1.2.0 with ozone.default.bucket.layout=OBS. Now, OM will set default bucket layout to OBS if a client creates a new bucket without specifying bucket layout argument.

case-1) step-1) An old_client_1.1.0 creates a new bucket BUCKET_01 in OM_1.2.0 and the bucket will be created successfully with default bucket layout OBS.

step-2) Now, new_client_1.2.0 performs o3fs/ofs operation on this new BUCKET_01 then it will throw validation error.

During upgrade there could be chances of both old_client_1.1.0 and new_client_1.2.0 co-exists, should we change the default value to ozone.default.bucket.layout=LEGACY ? With this the o3fs/ofs client will be happy and avoids validation errors.

This is the reason(ozone.default.bucket.layout=OBS) for many test failures in our CI.

@bharatviswa504, @mukul1987 welcome thoughts, thanks!

I have few questions.

1. Can old client talk to FSO buckets?

With this the o3fs/ofs client will be happy and avoids validation errors.

We are doing validation only in new client, not understood still how it can cause old clients an issue? (Because during upgrade based on the enable flag, we will change all the bucket types to OBS/LEGACY right?)

[bharatviswa504]

I think now I am understanding the reason behind the proposal of setting to LEGACY, so that in fs the check will pass, and we donot need to update tests. Is there any other reason other than tests update, it will cause issues in real deployments?

[rakeshadr]

@bharatviswa504 thanks for the quick response. Added my response here:

I think now I am understanding the reason behind the proposal of setting to LEGACY, so that in fs the check will pass, and we donot need to update tests.

Yes, this makes the existing test cases happy and these tests representsnew_client_1.2.0 FS operations. @JyotinderSingh please confirm.

Is there any other reason other than tests update, it will cause issues in real deployments?

@bharatviswa504 In real deployments during upgrade phase both old_client_1.1.0 and new_client_1.2.0 can co-exists. Adding the strict bucket layout validation will fail the ofs/o3fs based customer applications and they need to re-write it their application while upgrading to newer software and ensure that all FS operations has to be done only on FSO buckets, right?

Instead of changing the default value from OBS to LEGACY, I am thinking of introducing a client side flag

option-1) ozone.client.bucket.layout.validation=false. By default we can set it to false and once the cluster is fully upgraded customer can set it to true that will enable the o3fs/ofs validation check.

option-2) ozone.client.bucket.layouts.allowed=FSO,OBS. Here a comma-separated list of values where ofs/o3fs can do a validation check. Later after upgradation this can be set to FSO.

[rakeshadr]

Instead of changing the default value from OBS to LEGACY, I am thinking of introducing a client side flag

option-1) ozone.client.bucket.layout.validation=false. By default we can set it to false and once the cluster is fully upgraded customer can set it to true that will enable the o3fs/ofs validation check.

option-2) ozone.client.bucket.layouts.allowed=FSO,OBS. Here a comma-separated list of values where ofs/o3fs can do a validation check. Later after upgradation this can be set to FSO.

The client side validation will be handled in jira HDDS-5959 and will continue the discussion there.

[rakeshadr]

+1 LGTM.

Thank you @JyotinderSingh for the good work and will merge it once we get a clean CI report.

Thanks @mukul1987 , @bharatviswa504 , @aryangupta1998 for the reviews.