[HUDI-2296] pr flink support ConsistencyGuard plugin#3453
[HUDI-2296] pr flink support ConsistencyGuard plugin#3453gengxuhan wants to merge 2 commits intoapache:masterfrom
Conversation
|
flink support ConsistencyGuard plugin,When there is metadata latency in the file system,Submit CKP once, and then start a new instant. Because the commit cannot be seen, the data will be rolled back. @leesf |
| public static final ConfigOption<Boolean> CONSISTENCY_CHECK_ENABLED_PROP = ConfigOptions | ||
| .key("hoodie.consistency.check.enabled") | ||
| .booleanType() | ||
| .defaultValue(false) |
There was a problem hiding this comment.
Can we use shorter name for flink options: consistency.check.enabled
|
|
||
| public static final ConfigOption<Integer> INITIAL_CONSISTENCY_CHECK_INTERVAL_MS_PROP = ConfigOptions | ||
| .key("hoodie.consistency.check.initial_interval_ms") | ||
| .intType() |
| .withDescription("Amount of time (in ms) to wait, before checking for consistency after an operation on storage."); | ||
|
|
||
| public static final ConfigOption<Integer> MAX_CONSISTENCY_CHECK_INTERVAL_MS_PROP = ConfigOptions | ||
| .key("hoodie.consistency.check.max_interval_ms") |
| .withDescription("Maximum amount of time (in ms), to wait for consistency checking."); | ||
|
|
||
| public static final ConfigOption<Integer> MAX_CONSISTENCY_CHECKS_PROP = ConfigOptions | ||
| .key("hoodie.consistency.check.max_checks") |
|
@hudi-bot run azure |
|
@danny0405 hi , Can this PR be merged? |
| private void transitionState(HoodieInstant fromInstant, HoodieInstant toInstant, Option<byte[]> data) { | ||
| transitionState(fromInstant, toInstant, data, false); | ||
| transitionState(fromInstant, toInstant, data, this.metaClient.getConsistencyGuardConfig().isConsistencyCheckEnabled()); | ||
| } |
There was a problem hiding this comment.
Can you explain why the consistency check enabled then allows the duplicate files ?
There was a problem hiding this comment.
@danny0405 Because in the HoodieWrapperFileSystem file system, only createNewFile uses consistencyguard,
If not called, it will not take effect
yihua
added
writer-core
engine:flink
|
@danny0405 is this still relevant? If not, could we close the PR? |
|
Let's close it first. |
4 participants
Tips
What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.