HIVE-26603: Initiator does not initiate minor compaction for insert-o… #3648
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
okumin
pushed a commit
to okumin/hive
that referenced
this pull request
Oct 7, 2022
* Source Iceberg PR - Core: Remove deprecated APIs up to 0.13.0 * Revert "HIVE-25563: Iceberg table operations hang a long time if metadata is missing/corrupted (Adam Szita, reviewed by Marton Bod)" - applying instead Hive: Limit number of retries when metadata file is missing (apache#3379) This reverts commit 7b600fe. * Source Iceberg PR - Hive: Limit number of retries when metadata file is missing (apache#3379) * Source Iceberg PR - Hive: Fix RetryingMetaStoreClient for Hive 2.1 (apache#3403) * Source Iceberg PR - Switch from new HashMap to Maps.newHashMap (apache#3648) * Source Iceberg PR - Hive: HiveCatalog should remove HMS stats for certain engines based on config (apache#3652) - Use the Iceberg config property * Source Iceberg PR - Core: If status check fails, commit should be unknown (apache#3717) * Source Iceberg PR - Build: Add checkstyle rule for instantiating HashMap, HashSet, ArrayList (apache#3689) * Source Iceberg PR - Test: Make sure to delete temp folders (apache#3790) * Source Iceberg PR - API: Register existing tables in Iceberg HiveCatalog (apache#3851) * Source Iceberg PR - Hive: Make Iceberg table filter optional in HiveCatalog (apache#3908) * Source Iceberg PR - Core: Add reserved UUID Table Property and Expose in HMS. (apache#3914) * Source Iceberg PR - Hive: Known exception should not become CommitStateUnknownException (apache#4261) * Source Iceberg PR - Build: Add missing @OverRide annotations (apache#3654)
kgyrtkirk
added
tests failed
tests pending
tests unstable
and removed
tests pending
tests failed
labels
kgyrtkirk
added
tests pending
tests failed
tests passed
and removed
tests unstable
tests pending
tests failed
tests passed
labels
SourabhBadhya
force-pushed
the
HIVE-26603
branch
from
97cd311
to
4ee393a
Compare
|
Kudos, SonarCloud Quality Gate passed!
|
DongWei-4
pushed a commit
to DongWei-4/hive
that referenced
this pull request
Oct 28, 2022
* Source Iceberg PR - Core: Remove deprecated APIs up to 0.13.0 * Revert "HIVE-25563: Iceberg table operations hang a long time if metadata is missing/corrupted (Adam Szita, reviewed by Marton Bod)" - applying instead Hive: Limit number of retries when metadata file is missing (apache#3379) This reverts commit 7b600fe. * Source Iceberg PR - Hive: Limit number of retries when metadata file is missing (apache#3379) * Source Iceberg PR - Hive: Fix RetryingMetaStoreClient for Hive 2.1 (apache#3403) * Source Iceberg PR - Switch from new HashMap to Maps.newHashMap (apache#3648) * Source Iceberg PR - Hive: HiveCatalog should remove HMS stats for certain engines based on config (apache#3652) - Use the Iceberg config property * Source Iceberg PR - Core: If status check fails, commit should be unknown (apache#3717) * Source Iceberg PR - Build: Add checkstyle rule for instantiating HashMap, HashSet, ArrayList (apache#3689) * Source Iceberg PR - Test: Make sure to delete temp folders (apache#3790) * Source Iceberg PR - API: Register existing tables in Iceberg HiveCatalog (apache#3851) * Source Iceberg PR - Hive: Make Iceberg table filter optional in HiveCatalog (apache#3908) * Source Iceberg PR - Core: Add reserved UUID Table Property and Expose in HMS. (apache#3914) * Source Iceberg PR - Hive: Known exception should not become CommitStateUnknownException (apache#4261) * Source Iceberg PR - Build: Add missing @OverRide annotations (apache#3654)
DongWei-4
pushed a commit
to DongWei-4/hive
that referenced
this pull request
Oct 28, 2022
…nly tables (apache#3648). (Sourabh Badhya, reviewed by László Végh and Ayush Saxena)
dengzhhu653
pushed a commit
to dengzhhu653/hive
that referenced
this pull request
Dec 15, 2022
* Source Iceberg PR - Core: Remove deprecated APIs up to 0.13.0 * Revert "HIVE-25563: Iceberg table operations hang a long time if metadata is missing/corrupted (Adam Szita, reviewed by Marton Bod)" - applying instead Hive: Limit number of retries when metadata file is missing (apache#3379) This reverts commit 7b600fe. * Source Iceberg PR - Hive: Limit number of retries when metadata file is missing (apache#3379) * Source Iceberg PR - Hive: Fix RetryingMetaStoreClient for Hive 2.1 (apache#3403) * Source Iceberg PR - Switch from new HashMap to Maps.newHashMap (apache#3648) * Source Iceberg PR - Hive: HiveCatalog should remove HMS stats for certain engines based on config (apache#3652) - Use the Iceberg config property * Source Iceberg PR - Core: If status check fails, commit should be unknown (apache#3717) * Source Iceberg PR - Build: Add checkstyle rule for instantiating HashMap, HashSet, ArrayList (apache#3689) * Source Iceberg PR - Test: Make sure to delete temp folders (apache#3790) * Source Iceberg PR - API: Register existing tables in Iceberg HiveCatalog (apache#3851) * Source Iceberg PR - Hive: Make Iceberg table filter optional in HiveCatalog (apache#3908) * Source Iceberg PR - Core: Add reserved UUID Table Property and Expose in HMS. (apache#3914) * Source Iceberg PR - Hive: Known exception should not become CommitStateUnknownException (apache#4261) * Source Iceberg PR - Build: Add missing @OverRide annotations (apache#3654)
dengzhhu653
pushed a commit
to dengzhhu653/hive
that referenced
this pull request
Dec 15, 2022
…nly tables (apache#3648). (Sourabh Badhya, reviewed by László Végh and Ayush Saxena)
yeahyung
pushed a commit
to yeahyung/hive
that referenced
this pull request
Jul 20, 2023
…nly tables (apache#3648). (Sourabh Badhya, reviewed by László Végh and Ayush Saxena)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…nly tables
What changes were proposed in this pull request?
Initiator should initiate minor compaction for insert-only tables.
Why are the changes needed?
Initiator always initiates major compaction whenever it tries to initiate compaction on insert-only tables (MM tables). This could lead to significant work being done for compacting a large insert-only table every single time since major compaction would practically create a new base file & cleans all the delta files present.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit test