Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-48699][SQL] Refine collation API #47014

Closed
wants to merge 9 commits into from
Closed

[SPARK-48699][SQL] Refine collation API #47014

wants to merge 9 commits into from

Conversation

uros-db
Copy link
Contributor

@uros-db uros-db commented Jun 18, 2024 •
edited
Loading

What changes were proposed in this pull request?

  • All collation-related public API (i.e. CollationFactory, CollationAwareUTF8String, etc.) should generally use UTF8String instead of (Java) String.
  • Comparator for UTF8_LCASE collation should use UTF8String.binaryCompare instead of Java.compareTo.
  • Added ASCII “fast” path for Lower & Upper expressions.

Why are the changes needed?

  • Fix collation API.
  • Fix a bug in the UTF8_LCASE comparator.
  • Enhance performance for Lower & Upper expressions.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

  • Added more tests to CollationSupportSuite to verify that UTF8-based comparators indeed work properly.
  • Existing tests.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the SQL label Jun 18, 2024
@uros-db uros-db changed the title [WIP][SQL] Fix CollationFactory API [WIP][SQL] Improve collation API Jun 20, 2024
@uros-db uros-db changed the title [WIP][SQL] Improve collation API [SPARK-48699][SQL] Refine collation API Jun 24, 2024
mkaravel
mkaravel reviewed Jun 25, 2024
View reviewed changes
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java Outdated Show resolved Hide resolved
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java Outdated Show resolved Hide resolved
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java Show resolved Hide resolved
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java Show resolved Hide resolved
@uros-db uros-db requested a review from mkaravel June 25, 2024 17:59
mkaravel
mkaravel approved these changes Jun 25, 2024
View reviewed changes
Copy link
Contributor

@mkaravel mkaravel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
I left some comments about the TODO comments and also asking to add a couple more.
Please address these before merging.

common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Outdated Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Show resolved Hide resolved
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java Show resolved Hide resolved
@uros-db
Copy link
Contributor Author

uros-db commented Jun 26, 2024

@cloud-fan ready to merge

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in ec0ee86 Jun 26, 2024
@uros-db uros-db mentioned this pull request Jun 26, 2024
Closed
cloud-fan pushed a commit that referenced this pull request Jun 26, 2024
@uros-db @cloud-fan
[SPARK-48699][SQL][FOLLOWUP] Refine collation API
4cf5450 
### What changes were proposed in this pull request?
Following up on #47014, a small fix.

### Why are the changes needed?
CI is broken to due a conflict after merge.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Existing tests.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #47098 from uros-db/fix-spec.

Authored-by: Uros Bojanic <157381213+uros-db@users.noreply.github.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cloud-fan cloud-fan cloud-fan approved these changes

@mkaravel mkaravel mkaravel approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
3 participants
@uros-db @cloud-fan @mkaravel