[FLINK-25876] Implement overwrite in FlinkStoreCommitImpl #16
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tsreaper
changed the title
JingsongLi
reviewed
Feb 14, 2022
| public byte[] serialize(ManifestCommittable obj) throws IOException { | ||
| ByteArrayOutputStream out = new ByteArrayOutputStream(); | ||
| DataOutputViewStreamWrapper view = new DataOutputViewStreamWrapper(out); | ||
| view.writeInt(obj.uuid().length()); |
| } | ||
|
|
||
| @Nullable | ||
| private static ManifestFileMeta merge(List<ManifestFileMeta> metas, ManifestFile manifestFile) { |
| } | ||
| changesWithOverwrite.addAll(changes); | ||
|
|
||
| if (tryCommitOnce(changesWithOverwrite, hash, commitKind, latestSnapshotId)) { |
Contributor
There was a problem hiding this comment.
It seems that there is no need to check noConflictsOrFail again?
Contributor
Author
There was a problem hiding this comment.
Another job might be overwriting the same partition at the same time, so we still need to check this.
Contributor
There was a problem hiding this comment.
If I understand correctly, both checks are for the same snapshot, there can't be a new snapshot here.
| List<String> fieldNames = partitionType.getFieldNames(); | ||
| RowDataSerializer serializer = new RowDataSerializer(partitionType); | ||
| GenericRowData rowData = new GenericRowData(partitionType.getFieldCount()); | ||
| for (Map.Entry<String, String> entry : partition.entrySet()) { |
Contributor
There was a problem hiding this comment.
check that partition has no full partition keys
Contributor
There was a problem hiding this comment.
Here we can have the partial partition and we can add tests for it
JingsongLi
reviewed
Feb 15, 2022
| this.newFiles = new HashMap<>(); | ||
| this.compactBefore = new HashMap<>(); | ||
| this.compactAfter = new HashMap<>(); | ||
| this(UUID.randomUUID().toString(), new HashMap<>(), new HashMap<>(), new HashMap<>()); |
Contributor
There was a problem hiding this comment.
This is not work in the latest Flink master...
Contributor
There was a problem hiding this comment.
Let's see if we can restore the previous behavior
Alibaba-HZY
pushed a commit
to Alibaba-HZY/incubator-paimon
that referenced
this pull request
Apr 11, 2023
parent 2a62022 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226783 +0800 # 这是一个 13 个提交的组合。 parent 2a62022 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226730 +0800 # 这是一个 8 个提交的组合。tree 46928987599714b071489a7b6d4957049e6ded7a parent 2a62022 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 [core] Add streaming read from option (apache#778) # 这是提交说明 #8： [core] add test (apache#778) # 这是提交说明 apache#11： Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> # 这是提交说明 apache#12： Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> # 这是提交说明 apache#13： Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> # 这是提交说明 apache#15： Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> # 这是提交说明 apache#16： Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> # 这是提交说明 apache#17： Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> # 这是提交说明 apache#18： [core] add test (apache#778) # 这是提交说明 apache#19： merger sdda 这是一个 3 个提交的组合。 [core] commit1 (apache#778) [core] commit2 (apache#778) [core] commit3 (apache#778) [core] do commit 1(apache#778) [core] do commit 2(apache#778) # 这是提交说明 apache#20： sdda 这是一个 3 个提交的组合。 [core] commit1 (apache#778) [core] commit2 (apache#778) [core] commit3 (apache#778)
tub
pushed a commit
to tub/paimon
that referenced
this pull request
Jan 13, 2026
…readble_row_skip PRODENAB-149: skip unreadable records for all cdc writers
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overwrite is a useful transaction for batch jobs to completely update a partition for data correction. Currently FileStoreScanImpl doesn't implement this transaction so we need to implement that.