[Kernel] Change string comparing from UTF16 to UTF8 #3611
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ilicmarkodb
changed the title
ilicmarkodb
changed the title
ilicmarkodb
changed the title
vkorukanti
reviewed
Aug 28, 2024
Collaborator
vkorukanti
left a comment
vkorukanti
left a comment
There was a problem hiding this comment.
Thanks for catching this and fixing it.
|
|
||
| import io.delta.golden.GoldenTableUtils.goldenTablePath | ||
| import io.delta.kernel.exceptions.{InvalidTableException, KernelException, TableNotFoundException} | ||
| import io.delta.kernel.expressions.{Literal, ScalarExpression} |
Collaborator
There was a problem hiding this comment.
unrelated changes in this file?
Comment on lines
610
to
615
| ( | ||
| ofString("apples"), | ||
| ofString("oranges"), | ||
| ofString("apples"), | ||
| ofNull(StringType.STRING) | ||
| ), |
Collaborator
There was a problem hiding this comment.
same test case as the one above?
Comment on lines
616
to
621
| ( | ||
| ofString("abc"), | ||
| ofString("abcd"), | ||
| ofString("abc"), | ||
| ofNull(StringType.STRING) | ||
| ), |
Collaborator
There was a problem hiding this comment.
Suggested change
| ( | |
| ofString("abc"), | |
| ofString("abcd"), | |
| ofString("abc"), | |
| ofNull(StringType.STRING) | |
| ), | |
| (ofString("abc"), ofString("abcd"), ofString("abc"), ofNull(STRING)), |
Collaborator
There was a problem hiding this comment.
same for other test cases.
| ofString("\uD83C\uDF3C"), | ||
| ofString("\uFFFD"), | ||
| ofNull(StringType.STRING) | ||
| // scalastyle:on nonascii |
Collaborator
There was a problem hiding this comment.
are there more tests we could copy for Spark or Delta-Spark around other surrogate pairs and unsigned char comparison?
stefankandic
approved these changes
Aug 28, 2024
vkorukanti
approved these changes
Aug 28, 2024
vkorukanti
changed the title
vkorukanti
pushed a commit
to vkorukanti/delta
that referenced
this pull request
Aug 30, 2024
Changed string comparing from UTF16 to UTF8. This fixes comparison issues around the characters with surrogate pairs. Tests added to `DefaultExpressionEvaluatorSuite.scala`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Changed string comparing from UTF16 to UTF8. This fixes comparison issues around the characters with surrogate pairs.
How was this patch tested?
Tests added to
DefaultExpressionEvaluatorSuite.scala