Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-21228][SQL][BRANCH-2.2] InSet incorrect handling of structs #18563

Closed
wants to merge 1 commit into from

Conversation

Projects
None yet
3 participants
@bogdanrdc
Copy link
Contributor

commented Jul 7, 2017

What changes were proposed in this pull request?

This is backport of #18455
When data type is struct, InSet now uses TypeUtils.getInterpretedOrdering (similar to EqualTo) to build a TreeSet. In other cases it will use a HashSet as before (which should be faster). Similarly, In.eval uses Ordering.equiv instead of equals.

How was this patch tested?

New test in SQLQuerySuite.

@SparkQA

This comment has been minimized.

Copy link

commented Jul 7, 2017

Test build #79333 has finished for PR 18563 at commit fe09c9a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@bogdanrdc

This comment has been minimized.

Copy link
Contributor Author

commented Jul 8, 2017

@cloud-fan

This comment has been minimized.

Copy link
Contributor

commented Jul 8, 2017

thanks, merging to 2.2

asfgit pushed a commit that referenced this pull request Jul 8, 2017

[SPARK-21228][SQL][BRANCH-2.2] InSet incorrect handling of structs
## What changes were proposed in this pull request?

This is backport of #18455
When data type is struct, InSet now uses TypeUtils.getInterpretedOrdering (similar to EqualTo) to build a TreeSet. In other cases it will use a HashSet as before (which should be faster). Similarly, In.eval uses Ordering.equiv instead of equals.

## How was this patch tested?
New test in SQLQuerySuite.

Author: Bogdan Raducanu <bogdan@databricks.com>

Closes #18563 from bogdanrdc/SPARK-21228-BRANCH2.2.

@bogdanrdc bogdanrdc closed this Jul 12, 2017

MatthewRBruce added a commit to Shopify/spark that referenced this pull request Jul 31, 2018

[SPARK-21228][SQL][BRANCH-2.2] InSet incorrect handling of structs
## What changes were proposed in this pull request?

This is backport of apache#18455
When data type is struct, InSet now uses TypeUtils.getInterpretedOrdering (similar to EqualTo) to build a TreeSet. In other cases it will use a HashSet as before (which should be faster). Similarly, In.eval uses Ordering.equiv instead of equals.

## How was this patch tested?
New test in SQLQuerySuite.

Author: Bogdan Raducanu <bogdan@databricks.com>

Closes apache#18563 from bogdanrdc/SPARK-21228-BRANCH2.2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.