[HUDI-4724]Add function of skip the _rt suffix for read snapshot#6510
[HUDI-4724]Add function of skip the _rt suffix for read snapshot#6510linfey90 wants to merge 7 commits intoapache:masterfrom
Conversation
| public static final ConfigOption<Boolean> HIVE_SYNC_SKIP_RT_SUFFIX = ConfigOptions | ||
| .key("hive_sync.skip_rt_suffix") | ||
| .booleanType() | ||
| .defaultValue(false) |
There was a problem hiding this comment.
may you add sinceValue("0.12.1") to track in the doc when to use?
There was a problem hiding this comment.
Yes, I'd love to, but there's no since method. Any other suggestions?
| .withDocumentation("Skip the _ro suffix for Read optimized table, when registering"); | ||
| public static final ConfigProperty<String> HIVE_SKIP_RT_SUFFIX_FOR_READ_SNAPSHOT_TABLE = ConfigProperty | ||
| .key("hoodie.datasource.hive_sync.skip_rt_suffix") | ||
| .defaultValue("false") |
yihua
added
priority:medium
| this.snapshotTableName = tableName + SUFFIX_SNAPSHOT_TABLE; | ||
| this.snapshotTableName = config.getBoolean(HIVE_SKIP_RT_SUFFIX_FOR_READ_SNAPSHOT_TABLE) | ||
| ? tableName | ||
| : tableName + SUFFIX_SNAPSHOT_TABLE; |
There was a problem hiding this comment.
What's the situation if both skip_ro_suffix and skip_rt_suffix are true.
There was a problem hiding this comment.
Thanks for your reply.That's very thoughtful of you. Can we throw an exception in this case? Any suggestion do you have.
There was a problem hiding this comment.
We can throw exception if both options are valuable, but before do that, let's figure out the background why HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE is introduced ?
There was a problem hiding this comment.
During Hive query, first, we prefer to use the original table name, and second, we query the latest data.
| if (config.getBoolean(HIVE_SKIP_RT_SUFFIX_FOR_READ_SNAPSHOT_TABLE) | ||
| && config.getBoolean(HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE)) { | ||
| throw new HoodieHiveSyncException("can not be set both skip _rt and _ro are true at the same time!"); | ||
| } |
There was a problem hiding this comment.
can not be set both skip _rt and _ro are true
Can not skip _ro and _rt suffix both at the same time
|
hi @linfey90 What is your WeChat ID? I want to communicate with you. |
6 participants
Change Logs
add function of skip the _rt suffix for read snapshot when hive sync meta.
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level: none | low | medium | high
Choose one. If medium or high, explain what verification was done to mitigate the risks.
Contributor's checklist