fix: fix downgrade to not delete unintended partitions in MDT #14162
Conversation
github-actions
bot
added
size:M
|
PR desc please |
| + " or when using a custom Hoodie Concat Handle Implementation controlled by the config " + CONCAT_HANDLE_CLASS_NAME.key() | ||
| + ", enabling this config results in fallback to the default implementations if instantiation of the custom implementation fails"); | ||
|
|
||
| public static final ConfigProperty<Boolean> AUTO_DETECT_AND_DELETE_MDT_PARTITIONS = ConfigProperty |
There was a problem hiding this comment.
this is not end user config. we should remove this. we just need it as an additional argument to method calls. those originating from downgrade, dropIndex apis should set the argument value to false.
rest of the callers should leave it as true
There was a problem hiding this comment.
code path causing the issue is getting called from multiple places in the Upgrade/Downgrade code path which is making the approach of using additional argument to method calls very cumbersome.
For e.g rollbackFailedWritesAndCompact during downgrade also calls tableMetadata.getMetadataWriter which is the method that's causing the issue
| val cowTableName = "hudi-v9-table-mdt-validation-cow" | ||
| val morTableName = "hudi-v9-table-mdt-validation-mor" | ||
|
|
||
| val cowTableBasePath = basePath + "/" + cowTableName |
There was a problem hiding this comment.
can you help me remind why do we need a test fixture for this?
a functional test should suffice right?
we can create a v9 table and then downgrade to v8. and just validate the MDT partitions
There was a problem hiding this comment.
Makes sense, fixed the test
|
also, why the patch is in draft state? |
|
also, looks like you need to check CI failures as well. |
vamsikarnika
changed the title
Made it ready for review, was working on UT |
vamsikarnika
changed the title
vamsikarnika
force-pushed
the
fix_rli_delete_partitions
branch
from
8f3c2c2 to
11b94c6
Compare
...-datasource/hudi-spark/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java
Show resolved
Hide resolved
...-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java
Outdated
Show resolved
Hide resolved
nsivabalan
force-pushed
the
fix_rli_delete_partitions
branch
from
104cfdb to
64b6e81
Compare
|
reviewed, pushed out a commit to address my own feedback. |
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/HoodieJavaTable.java
Show resolved
Hide resolved
|
pushed another commit to remove the new write config and use arguments. |
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
Outdated
Show resolved
Hide resolved
|
--------- Co-authored-by: Vamsi <vamsi@onehouse.ai> Co-authored-by: sivabalan <n.siva.b@gmail.com>
4 participants
Describe the issue this Pull Request addresses
Fixes an issue where MDT partitions were being unintentionally deleted during downgrade operations
Summary and Changelog
Downgrade operation doesn't drop any partitions.
Impact
MDT partitions are not deleted during downgrade/upgrade operation
Risk Level
low
Documentation Update
NA
Contributor's checklist