HDDS-5094. [FSO] Fail OM startup when turn on prefix layout with old buckets #2151

rakeshadr · 2021-04-12T15:07:45Z

What changes were proposed in this pull request?

Start an existing OM using new PREFIX configuration layout format, which(OM) already contains old buckets.

Scenario:

Start cluster with configs,
OZONE-SITE.XML_ozone.om.enable.filesystem.paths=true
OZONE-SITE.XML_ozone.om.metadata.layout=simple
create /vol1/bucket1/dir1/dir2/key1
Stop OM
Update config OZONE-SITE.XML_ozone.om.metadata.layout=prefix
Start OM.

Output: Failed to start OM as there is a bucket with old layout format.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5094

How was this patch tested?

Added unit test case.

bharatviswa504 · 2021-04-12T15:55:05Z

@rakeshadr
I think with this approach, a cluster upgraded from an older version and if someone want to try a new layout version format, they cannot use it. IMHO, I think we should support older buckets read/write, and provide the capability to support users to try out the new format on the newly created buckets.

One Idea here is to persist format layout in bucket metadata and use that. And also iterate the buckets(Even now we iterate buckets during startup) and set layout version of the buckets to older format layout during startup if the format is not set. (I think this approach has been discussed during initial meetings)

bharatviswa504

I have one comment the approach/solution taken to solve this issue.

rakeshadr · 2021-04-12T17:13:43Z

@rakeshadr
I think with this approach, a cluster upgraded from an older version and if someone want to try a new layout version format, they cannot use it. IMHO, I think we should support older buckets read/write, and provide the capability to support users to try out the new format on the newly created buckets.

One Idea here is to persist format layout in bucket metadata and use that. And also iterate the buckets(Even now we iterate buckets during startup) and set layout version of the buckets to older format layout during startup if the format is not set. (I think this approach has been discussed during initial meetings)

Thanks @bharatviswa504 for bringing the point. I think, I haven't explain it clearly about the purpose of this patch. I will try to add more details here:

Yes, I have persisted the FSO layout in bucketInfo.getMetadata() (bucket level property) to differentiate older buckets and newer buckets. As OM is persisting the layout in the bucket level the feature can be extended to support both older and newer buckets. Please see rename and delete APIs, the older and newer buckets are implemented in that fashion. I have plans to support all the cases gracefully, I hope my comment in the jira conveys the same thoughts.

The purpose of this patch is to validate and fail during OM startup till I support and implement all the cases incrementally(phase by phase), then will remove this check. This means that, now in phase-1 the new feature is allowed only in a fresh new cluster. Later, I agree to support this case in follow-up tasks, existing old key(s) should be visible in prefix based OM layout.

elek

Thanks, the quick fix @rakeshadr.

This patch seems to disable the usage of this feature in any existing cluster, and it can be used only in new cluster. It seems to be a quite huge limitation and I would suggest to mention in at least in the doc: if somebody creates at least one bucket this value can not be changed anymore.
Can you please confirm if this check works well when the metadata is empty? I may be wrong (I didn't test) but I think if I create buckets with existing code (no metadata) and update to this branch and set the layout to 'simple' the cluster couldn't be started. Fix me if I am wrong.
This validation won't work if somebody enables only the prefix-layout without the fs path support. What is the expected behavior in this case? Do we need any validation to avoid such case?

elek · 2021-04-13T12:58:03Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

@@ -1115,6 +1118,9 @@ public void start() throws IOException {
    getOMMetadataLayout();

    metadataManager.start(configuration);
+
+    validatesBucketLayoutMismatches();


Can it be checked before starting the ratis/RPC servers? Seems to be more safe IMHO...

Can it be checked before starting the ratis/RPC servers? Seems to be more safe IMHO...

Thanks @elek for the comment. IMHO to maintain the existing order of starting of all the services as it is and don't like to impact existing OM startup behavior due to this temporary validation logic, by default the feature is disabled and there is no impact to the existing users/clusters. Like I said the contributors(dev team) will be actively working to support older[simple] buckets in prefix layout and will remove this validation logic later.

impact existing OM startup behavior due to this temporary validation logic

Thanks the answer @rakeshadr. Can you please help me to understand how can moving this line upper would affect OM start behavior?

As far as I understand this check is only applied when OzoneManagerRatisUtils.isBucketFSOptimized is true. I just asked if this new (!) check would be more safe to do before the ratis initialization as additional actions may happen before this safety check stops the clusters.

@elek Yes, its validating the bucket metadata only when OzoneManagerRatisUtils.isBucketFSOptimized is true. IIUC, you are suggesting to do the modification like below in OzoneManager#start() function, right? .

I have tried this but my test fail as I could see validation logic reads bucketMetadata successfully only if I keep it after metadataManager.start(configuration); function.
The validation logic reads bucket table data using iterator = metadataManager.getBucketTable().iterator();,

getOMMetadataLayout(); validatesBucketLayoutMismatches(); // Start Ratis services if (omRatisServer != null) { omRatisServer.start(); } metadataManager.start(configuration);

@elek I hope you agree to move the metadataManager.start(configuration) call upwards before the ratis server start ?

OzoneManager#start() LOG.info(buildRpcServerStartMessage("OzoneManager RPC server", omRpcAddress)); metadataManager.start(configuration); getOMMetadataLayout(); validatesBucketLayoutMismatches(); // Start Ratis services if (omRatisServer != null) { omRatisServer.start(); } startSecretManagerIfNecessary();

@elek I've modified the startup steps in latest (2nd) commit -

Validate Invalid Configuration(layout PREFIX and ozone.om.enable.filesystem.paths=false) at the beginning

Moved validatesBucketLayoutMismatches before ratis server.

Hope this looks fine to you. Thanks!

rakeshadr · 2021-04-19T05:24:06Z

This patch seems to disable the usage of this feature in any existing cluster, and it can be used only in new cluster. It seems to be a quite huge limitation and I would suggest to mention in at least in the doc: if somebody creates at least one bucket this value can not be changed anymore.

Sure, I will update the wiki page and apache docs. "The feature is disabled by default and can be used only in new cluster".

Can you please confirm if this check works well when the metadata is empty? I may be wrong (I didn't test) but I think if I create buckets with existing code (no metadata) and update to this branch and set the layout to 'simple' the cluster couldn't be started. Fix me if I am wrong.

Your previous question has the answer - "it can be used only in new cluster". A new cluster will not have any volumes. Simple answer is, during cluster startup the validation logic will check all the existing buckets, if any and then will fail the OM startup if there are any older buckets[simple layout].

This validation won't work if somebody enables only the prefix-layout without the fs path support. What is the expected behavior in this case? Do we need any validation to avoid such case?

Please refer isFSOptimizedBucket logic. Here it checks both the properties and FSO requires both to be set(ozone.om.metadata.layout=PREFIX and ozone.om.enable.filesystem.paths=true).

rakeshadr · 2021-04-19T05:25:50Z

Thanks @elek for the review comments.

bharatviswa504 · 2021-04-19T06:07:03Z

@rakeshadr
I think with this approach, a cluster upgraded from an older version and if someone want to try a new layout version format, they cannot use it. IMHO, I think we should support older buckets read/write, and provide the capability to support users to try out the new format on the newly created buckets.
One Idea here is to persist format layout in bucket metadata and use that. And also iterate the buckets(Even now we iterate buckets during startup) and set layout version of the buckets to older format layout during startup if the format is not set. (I think this approach has been discussed during initial meetings)

Thanks @bharatviswa504 for bringing the point. I think, I haven't explain it clearly about the purpose of this patch. I will try to add more details here:

Yes, I have persisted the FSO layout in bucketInfo.getMetadata() (bucket level property) to differentiate older buckets and newer buckets. As OM is persisting the layout in the bucket level the feature can be extended to support both older and newer buckets. Please see rename and delete APIs, the older and newer buckets are implemented in that fashion. I have plans to support all the cases gracefully, I hope my comment in the jira conveys the same thoughts.

The purpose of this patch is to validate and fail during OM startup till I support and implement all the cases incrementally(phase by phase), then will remove this check. This means that, now in phase-1 the new feature is allowed only in a fresh new cluster. Later, I agree to support this case in follow-up tasks, existing old key(s) should be visible in prefix based OM layout.

Thank You @rakeshadr for the clear explanation and offline discussion. I am fine with this, if we are planning to address them in furthere Jiras.

elek · 2021-04-19T09:57:12Z

Simple answer is, during cluster startup the validation logic will check all the existing buckets, if any and then will fail the OM startup if there are any older buckets[simple layout].

Fix me if I am wrong, but the current check doesn't handle case when bucketMetadata is empty (old data). For example if I set layout prefix to simple explicitly (because I have old data) the cluster will be failed even if it's a valid scenario.

+  public static boolean isFSOptimizedBucket(
+      Map<String, String> bucketMetadata) {
+    // layout 'PREFIX' represents optimized FS path
+    boolean metadataLayoutEnabled =
+        org.apache.commons.lang3.StringUtils.equalsIgnoreCase(
+            OMConfigKeys.OZONE_OM_METADATA_LAYOUT_PREFIX,
+            bucketMetadata
+                .get(OMConfigKeys.OZONE_OM_METADATA_LAYOUT));
+
+    boolean fsEnabled =
+        Boolean.parseBoolean(bucketMetadata
+            .get(OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS));
+
+    return metadataLayoutEnabled && fsEnabled;
+  }

Please refer isFSOptimizedBucket logic. Here it checks both the properties and FSO requires both to be set(ozone.om.metadata.layout=PREFIX and ozone.om.enable.filesystem.paths=true).

Thanks, will check it. For your information: during my test I was able to create buckets with layout=PREFIX and paths=false. Will try to reproduce the problem.

elek · 2021-04-19T10:10:13Z

I can confirm that I can create buckets where PREFIX is enabled but filesystem.paths is not:

bash-4.2$ ozone sh volume create /vol1
bash-4.2$ ozone sh bucket create /vol1/bucket1
bash-4.2$ ozone sh bucket info /vol1/bucket1     
{
  "metadata" : {
    "ozone.om.metadata.layout" : "PREFIX",
    "ozone.om.enable.filesystem.paths" : "false"
  },
  "volumeName" : "vol1",
  "name" : "bucket1",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 0,
  "usedNamespace" : 0,
  "creationTime" : "2021-04-19T10:06:25.059Z",
  "modificationTime" : "2021-04-19T10:06:25.059Z",
  "encryptionKeyName" : null,
  "sourceVolume" : null,
  "sourceBucket" : null,
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1
}
bash-4.2$ cat /etc/hadoop/ozone-site.xml | grep paths
bash-4.2$ cat /etc/hadoop/ozone-site.xml | grep layout
<property><name>ozone.om.metadata.layout</name><value>PREFIX</value></property>
bash-4.2$

If it's a supported case we need to improve the validation of this patch. If it's not, we should open a a new issue for validation and deny this mis-configuration as it can cause problems.

rakeshadr · 2021-04-19T11:07:23Z

I can confirm that I can create buckets where PREFIX is enabled but filesystem.paths is not:

bash-4.2$ ozone sh volume create /vol1
bash-4.2$ ozone sh bucket create /vol1/bucket1
bash-4.2$ ozone sh bucket info /vol1/bucket1     
{
  "metadata" : {
    "ozone.om.metadata.layout" : "PREFIX",
    "ozone.om.enable.filesystem.paths" : "false"
  },
  "volumeName" : "vol1",
  "name" : "bucket1",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 0,
  "usedNamespace" : 0,
  "creationTime" : "2021-04-19T10:06:25.059Z",
  "modificationTime" : "2021-04-19T10:06:25.059Z",
  "encryptionKeyName" : null,
  "sourceVolume" : null,
  "sourceBucket" : null,
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1
}
bash-4.2$ cat /etc/hadoop/ozone-site.xml | grep paths
bash-4.2$ cat /etc/hadoop/ozone-site.xml | grep layout
<property><name>ozone.om.metadata.layout</name><value>PREFIX</value></property>
bash-4.2$

If it's a supported case we need to improve the validation of this patch. If it's not, we should open a a new issue for validation and deny this mis-configuration as it can cause problems.

Thanks for the confirmation @elek with a test case. Instead of failing with an error, presently OM proceeds silently with default metadata layout by making isBucketFSOptimized flag to false. I agree to fail the startup and make it visible to everyone. I will include this check along with this OM start up validation patch.

rakeshadr · 2021-04-19T11:33:23Z

Simple answer is, during cluster startup the validation logic will check all the existing buckets, if any and then will fail the OM startup if there are any older buckets[simple layout].

Fix me if I am wrong, but the current check doesn't handle case when bucketMetadata is empty (old data). For example if I set layout prefix to simple explicitly (because I have old data) the cluster will be failed even if it's a valid scenario.
+  public static boolean isFSOptimizedBucket(
+      Map<String, String> bucketMetadata) {
+    // layout 'PREFIX' represents optimized FS path
+    boolean metadataLayoutEnabled =
+        org.apache.commons.lang3.StringUtils.equalsIgnoreCase(
+            OMConfigKeys.OZONE_OM_METADATA_LAYOUT_PREFIX,
+            bucketMetadata
+                .get(OMConfigKeys.OZONE_OM_METADATA_LAYOUT));
+
+    boolean fsEnabled =
+        Boolean.parseBoolean(bucketMetadata
+            .get(OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS));
+
+    return metadataLayoutEnabled && fsEnabled;
+  }

I'm adding test scenarios explicitly with config values to understand it better, please feel free to add if I missed anything. Thanks!

Scenario-1) Created a bucket with older cluster. Here bucketMetadata is empty.

Test Step-1) Stops OM server and updated configs ozone.om.metadata.layout=PREFIX and ozone.om.enable.filesystem.paths=true.

Test Step-2) Starts OM, validate isFSOptimizedBucket and metadataLayoutEnabled will be false.

Test Result: OM startup will fail.

Scenario-2) Created a bucket with older cluster. Here bucketMetadata is empty.

Test Step-1) Stops OM server and updated configs ozone.om.metadata.layout=SIMPLE and ozone.om.enable.filesystem.paths=true.

Test Step-2) Starts OM

Test Result: OM won't validate the bucketmetadata layout as the cluster level layout is SIMPLE. It will start successfully with prefix feature disabled.

Please add you expectations, thanks @elek

elek · 2021-04-21T08:29:53Z

Instead of failing with an error, presently OM proceeds silently with default metadata layout by making isBucketFSOptimized flag to false.

It might be better to throw an exception in case of misconfiguration. the isBucketFSOptimized can be false when I configure PREFIX + fs.enabled = false, but the wrong metadata was saved to the key:

bash-4.2$ ozone sh bucket info /vol1/bucket1     
{
  "metadata" : {
    "ozone.om.metadata.layout" : "PREFIX",
    "ozone.om.enable.filesystem.paths" : "false

Even better: it would be great to use just the ozone.om.metadata.layout and always turn on ozone.om.enable.filesystem.paths implicitly as it seems to be a strong requirement.

elek · 2021-04-21T08:33:18Z

Scenario-2) Created a bucket with older cluster. Here bucketMetadata is empty.

Test Step-1) Stops OM server and updated configs ozone.om.metadata.layout=SIMPLE and ozone.om.enable.filesystem.paths=true.

Test Step-2) Starts OM

Test Result: OM won't validate the bucketmetadata layout as the cluster level layout is SIMPLE. It will start successfully with prefix feature disabled.

I think it's better to always validate, because we have this scenario, too:

Scenarion-3) Created a bucket with the new cluster and layout=PREFIX. bucketMetadata is filled

Test Step-1) Stops OM server and updated configs ozone.om.metadata.layout=SIMPLE and ozone.om.enable.filesystem.paths=true.

Test Step-2) Starts OM

Test Result: OM should validate the layouts and fail as SIMPLE and PREFIX couldn't be handled in the same system (today).

elek · 2021-04-21T08:35:22Z

BTW, can we test it without using MiniOzoneCluster? Seems to be way more faster which would enable us the test more scenarios. I can be wrong, but it seems to be easy as we test a very well separated local check (we need bucket metadata + the checker...)

rakeshadr · 2021-04-21T09:17:14Z

Instead of failing with an error, presently OM proceeds silently with default metadata layout by making isBucketFSOptimized flag to false.

It might be better to throw an exception in case of misconfiguration. the isBucketFSOptimized can be false when I configure PREFIX + fs.enabled = false, but the wrong metadata was saved to the key:
bash-4.2$ ozone sh bucket info /vol1/bucket1     
{
  "metadata" : {
    "ozone.om.metadata.layout" : "PREFIX",
    "ozone.om.enable.filesystem.paths" : "false
Even better: it would be great to use just the ozone.om.metadata.layout and always turn on ozone.om.enable.filesystem.paths implicitly as it seems to be a strong requirement.

Thanks @elek for the clear information, IMHO to throw exception in case of invalid configuration and fail OM startup. I've updated the PR with this behavior.

rakeshadr · 2021-04-21T09:18:45Z

Scenarion-3) Created a bucket with the new cluster and layout=PREFIX. bucketMetadata is filled

Test Step-1) Stops OM server and updated configs ozone.om.metadata.layout=SIMPLE and ozone.om.enable.filesystem.paths=true.

Test Step-2) Starts OM

Test Result: OM should validate the layouts and fail as SIMPLE and PREFIX couldn't be handled in the same system (today).

Thanks @elek for pointing out this case. I have added this case in the latest commit.

rakeshadr · 2021-04-21T09:27:42Z

BTW, can we test it without using MiniOzoneCluster? Seems to be way more faster which would enable us the test more scenarios. I can be wrong, but it seems to be easy as we test a very well separated local check (we need bucket metadata + the checker...)

Thanks a lot @elek for the suggestion. Meantime, I've modified existing unit test case by covering the cases that we discussed in this PR but I am still using MiniOzoneCluster and would like to get your feedback on this. I am doing start/stop only OM to test various cases. I have ran this unit test multiple times(20 times) locally in my env and I could see that each run took only <=5secs. Hope this make sense to you!

FYI, I'm also having future plans to create MiniOMCluster by mocking other services except OM, especially for the PREFIX/SIMPLE based integration-tests(applicable tests). IMHO, that will save some additional cluster startup time. I will also revisit this unit test along with that task. I will raise a jira task to track it, in some time.

rakeshadr · 2021-04-26T05:20:39Z

Hi @elek, I've updated the PR based on your comment. Can you please review it again, thanks!

elek

Thanks, @rakeshadr the update. Validation logic looks good to me. 👍

I have a few -- very small -- nit comments (and I assume that some unit test changes supposed to be part of an integration test related patch not this one)

I have ran this unit test multiple times(20 times) locally in my env and I could see that each run took only <=5secs. Hope this make sense to you!

It seems you have a powerful machine. On CI server it's:

2021-04-26T03:50:12.2809634Z [INFO] Running org.apache.hadoop.ozone.om.TestOMStartupWithLayout
2021-04-26T03:50:45.3975735Z [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.113 s - in org.apache.hadoop.ozone.om.TestOMStartupWithLayout

But I think moving the check to the beginning is helped, and I didn't consider that in most of the cases the MiniOzoneCluster is not required to be started (as it should be failing) So let's keep it as is for now...

I'm also having future plans to create MiniOMCluster by mocking other services except OM, especially for the PREFIX/SIMPLE based integration-tests

👍 It sounds like a great idea. I like it.

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

...egration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystemWithFSO.java

...ne/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystemWithFSO.java

...ntegration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileInterfacesWithFSO.java

rakeshadr

Thanks again @elek for the review comments. Please find my reply and I will update the patch soon.

…buckets

…ne/om/OzoneManager.java Co-authored-by: Elek, Márton <elek@users.noreply.github.com>

rakeshadr · 2021-04-29T03:05:48Z

I have one comment the approach/solution taken to solve this issue.

Marked this as addressed based on @bharatviswa504's reply - comment link here

@bharatviswa504

Marked this as addressed based on @bharatviswa504's reply - #2151 (comment)

rakeshadr · 2021-04-29T03:09:04Z

@elek I've addressed your comments. Can you please review it again, thanks!

elek

+1, thanks, (the continuous) update and patience.

rakeshadr · 2021-04-30T07:23:47Z

Thanks a lot @elek, @bharatviswa504, @mukul1987 for the detailed reviews. After several CI runs, finally got a clean build report. I'm merging it to the branch.

rakeshadr requested review from mukul1987 and elek April 12, 2021 15:16

rakeshadr changed the title ~~HDDS-5094. [FSO] Existing keys become unavailable after turning on prefix layout~~ HDDS-5094. [FSO] Fail OM startup when turn on prefix layout with old buckets Apr 12, 2021

bharatviswa504 previously requested changes Apr 12, 2021

View reviewed changes

elek reviewed Apr 13, 2021

View reviewed changes

rakeshadr force-pushed the HDDS-5094 branch from abd9c7f to 5a04a53 Compare April 21, 2021 09:14

rakeshadr force-pushed the HDDS-5094 branch from fe33943 to 1916694 Compare April 25, 2021 08:50

elek reviewed Apr 26, 2021

View reviewed changes

rakeshadr commented Apr 26, 2021

View reviewed changes

rakeshadr and others added 4 commits April 28, 2021 21:07

HDDS-5094. [FSO] Fail OM startup when turn on prefix layout with old …

105db15

…buckets

Updated comments

bbf170b

Update hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozo…

cb5af48

…ne/om/OzoneManager.java Co-authored-by: Elek, Márton <elek@users.noreply.github.com>

Fixed review comments - changed to IAException and modified var name

3d3639b

rakeshadr force-pushed the HDDS-5094 branch from 953ac63 to 3d3639b Compare April 28, 2021 15:38

trigger new CI check

54d3b6c

rakeshadr added 4 commits April 29, 2021 16:26

Added unsupported config check

e2af09a

Replaced strings with constants

23528e3

trigger new CI check

24df3e4

trigger new CI check

c7c246b

elek approved these changes Apr 29, 2021

View reviewed changes

rakeshadr added 2 commits April 30, 2021 08:24

trigger new CI check

a14956b

trigger new CI check

e04d557

rakeshadr merged commit 3c82503 into apache:HDDS-2939 Apr 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-5094. [FSO] Fail OM startup when turn on prefix layout with old buckets #2151

HDDS-5094. [FSO] Fail OM startup when turn on prefix layout with old buckets #2151

rakeshadr commented Apr 12, 2021 •

edited

Loading

bharatviswa504 commented Apr 12, 2021 •

edited

Loading

bharatviswa504 left a comment

rakeshadr commented Apr 12, 2021 •

edited

Loading

elek left a comment

elek Apr 13, 2021

rakeshadr Apr 19, 2021

elek Apr 19, 2021

rakeshadr Apr 19, 2021

rakeshadr Apr 19, 2021

rakeshadr Apr 21, 2021

rakeshadr commented Apr 19, 2021

rakeshadr commented Apr 19, 2021

bharatviswa504 commented Apr 19, 2021

elek commented Apr 19, 2021 •

edited

Loading

elek commented Apr 19, 2021

rakeshadr commented Apr 19, 2021

rakeshadr commented Apr 19, 2021

elek commented Apr 21, 2021

elek commented Apr 21, 2021

elek commented Apr 21, 2021

rakeshadr commented Apr 21, 2021

rakeshadr commented Apr 21, 2021

rakeshadr commented Apr 21, 2021 •

edited

Loading

rakeshadr commented Apr 26, 2021

elek left a comment

rakeshadr left a comment

rakeshadr commented Apr 29, 2021

rakeshadr commented Apr 29, 2021

elek left a comment

rakeshadr commented Apr 30, 2021

HDDS-5094. [FSO] Fail OM startup when turn on prefix layout with old buckets #2151

HDDS-5094. [FSO] Fail OM startup when turn on prefix layout with old buckets #2151

Conversation

rakeshadr commented Apr 12, 2021 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

bharatviswa504 commented Apr 12, 2021 • edited Loading

bharatviswa504 left a comment

Choose a reason for hiding this comment

rakeshadr commented Apr 12, 2021 • edited Loading

elek left a comment

Choose a reason for hiding this comment

elek Apr 13, 2021

Choose a reason for hiding this comment

rakeshadr Apr 19, 2021

Choose a reason for hiding this comment

elek Apr 19, 2021

Choose a reason for hiding this comment

rakeshadr Apr 19, 2021

Choose a reason for hiding this comment

rakeshadr Apr 19, 2021

Choose a reason for hiding this comment

rakeshadr Apr 21, 2021

Choose a reason for hiding this comment

rakeshadr commented Apr 19, 2021

rakeshadr commented Apr 19, 2021

bharatviswa504 commented Apr 19, 2021

elek commented Apr 19, 2021 • edited Loading

elek commented Apr 19, 2021

rakeshadr commented Apr 19, 2021

rakeshadr commented Apr 19, 2021

elek commented Apr 21, 2021

elek commented Apr 21, 2021

elek commented Apr 21, 2021

rakeshadr commented Apr 21, 2021

rakeshadr commented Apr 21, 2021

rakeshadr commented Apr 21, 2021 • edited Loading

rakeshadr commented Apr 26, 2021

elek left a comment

Choose a reason for hiding this comment

rakeshadr left a comment

Choose a reason for hiding this comment

rakeshadr commented Apr 29, 2021

rakeshadr commented Apr 29, 2021

elek left a comment

Choose a reason for hiding this comment

rakeshadr commented Apr 30, 2021

rakeshadr commented Apr 12, 2021 •

edited

Loading

bharatviswa504 commented Apr 12, 2021 •

edited

Loading

rakeshadr commented Apr 12, 2021 •

edited

Loading

elek commented Apr 19, 2021 •

edited

Loading

rakeshadr commented Apr 21, 2021 •

edited

Loading