-
Notifications
You must be signed in to change notification settings - Fork 4.8k
HIVE-29028: Iceberg: Implement auto compaction #5886
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
7c50524
to
b75da71
Compare
b75da71
to
05f666a
Compare
05f666a
to
c6cbb86
Compare
c6cbb86
to
dc06a7c
Compare
plenty of reactions, but no reviews :) |
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
Outdated
Show resolved
Hide resolved
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java
Outdated
Show resolved
Hide resolved
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/IcebergInitiator.java
Outdated
Show resolved
Hide resolved
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/InitiatorBase.java
Outdated
Show resolved
Hide resolved
0023436
to
2830bda
Compare
2541f2d
to
4af67bc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly +1. I left some minor comments
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/TableOptimizer.java
Outdated
Show resolved
Hide resolved
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/TableOptimizer.java
Outdated
Show resolved
Hide resolved
4af67bc
to
c5290ea
Compare
c5290ea
to
77bc8d7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addressing the remaining issues!
|
Thanks, @deniskuzZ and @okumin for code reviews! |
What changes were proposed in this pull request?
Iceberg auto compaction. In this initial version,
IcebergInitiator
remembers the last checked snapshot id of all Iceberg tables and it compares the current snapshot id to the cached snapshot id to determine if further evaluation is needed.Why are the changes needed?
Currently, Iceberg compaction can only be triggered manually.
This PR introduces automatic Iceberg compaction which periodically checks all Iceberg tables and schedules compactions when necessary.
Does this PR introduce any user-facing change?
No
How was this patch tested?
New unit tests, precommit tests.