Core: Fanout equality/position delete writer #7313
Conversation
Fokko
changed the title
| * An position delete writer capable of writing to multiple specs and partitions that keeps position | ||
| * delete writer for each seen spec/partition pair open until this writer is closed. | ||
| */ | ||
| public class FanoutPositionDeleteWriter<T> |
There was a problem hiding this comment.
I doubt a fanout position delete writer is useful as the spec requires all position deletes within a file to be sorted by file name and position. That means the external process must at least locally order the incoming data. If a local sort is required, then we can include both spec ID and partition in the sort, which allows us to to use a clustered writer.
There was a problem hiding this comment.
Thanks! Your viewpoint is very correct. Actually, we only use FanoutEqualityDeleteWriter, I mistakenly believe that a equality deletes and position deletes are similar.
| * An equality delete writer capable of writing to multiple specs and partitions that keeps equality | ||
| * delete writer for each seen spec/partition pair open until this writer is closed. | ||
| */ | ||
| public class FanoutEqualityDeleteWriter<T> extends FanoutWriter<T, DeleteWriteResult> { |
There was a problem hiding this comment.
We would need to add tests, just like we have for FanoutDataWriter.
There was a problem hiding this comment.
We would need to add tests, just like we have for
FanoutDataWriter.
Okay, I'll try adding it
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
2 participants
Refer to clustered writer