HIVE-28798: Bucket Map Join partially using partition transforms - #5670
Merged
Conversation
okumin
force-pushed
the
HIVE-28798-partial-bucket
branch
from
March 4, 2025 02:57
ec6d94c to
1bc1871
Compare
okumin
commented
Mar 4, 2025
| (103, null, 'val_103', 18), | ||
| (104, '1004', null, 19), | ||
| (105, '1005', 'val_105', 20), | ||
| (null, null, null, 21); |
Contributor
Author
There was a problem hiding this comment.
I added many records so that the big table detection always picks up src_small
okumin
changed the title
okumin
marked this pull request as ready for review
March 4, 2025 03:10
okumin
commented
Mar 4, 2025
| newBucketFunctions.add(null); | ||
| break; | ||
| } | ||
| Optional<CustomBucketFunction> newBucketFunction = bucketFunction.select(retainedColumns); |
Contributor
Author
There was a problem hiding this comment.
I finally want to implement Hive's v1 and v2 bucketing as CustomBucketFunctions to unify the logic
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
okumin
force-pushed
the
HIVE-28798-partial-bucket
branch
from
March 4, 2025 23:51
1bc1871 to
14b8c95
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
okumin
force-pushed
the
HIVE-28798-partial-bucket
branch
from
April 3, 2025 14:09
14b8c95 to
b05bbe5
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
asf-ci-hive
added
tests passed
tests pending
tests unstable
and removed
tests passed
tests pending
tests unstable
labels
okumin
force-pushed
the
HIVE-28798-partial-bucket
branch
from
July 13, 2025 12:18
8468201 to
3d22f8b
Compare
asf-ci-hive
added
tests failed
tests pending
tests unstable
and removed
tests pending
tests failed
tests unstable
labels
|
Contributor
Author
|
I rebased this branch since it is too obsolete. Now, CI is green. |
Contributor
Author
|
@ngsg Thanks for your thorough review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR updates
OpTraitsRulesProcFactory.SelectRuleto propagate bucketing information when the source is an Iceberg table, and only a subset of bucketing columns is used.https://issues.apache.org/jira/browse/HIVE-28798
Why are the changes needed?
For better performance. Iceberg's transform spec allows us to bucketize multiple columns separately, e.g., stored in
/warehouse/db/table/data/key1=3/key2=5. Hive's one encodes a set of all bucketing columns into a single integer.Does this PR introduce any user-facing change?
No. The query plan can change, but BMJ on Iceberg has not been released yet.
Is the change a dependency upgrade?
No
How was this patch tested?
I updated
iceberg_bucket_map_join_8.qso that it includes various combinations.