Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-22464] [SQL] No pushdown for Hive metastore partition predicates containing null-safe equality #19682

Closed
wants to merge 3 commits into from

Conversation

Projects
None yet
4 participants
@gatorsmile
Copy link
Member

commented Nov 7, 2017

What changes were proposed in this pull request?

<=> is not supported by Hive metastore partition predicate pushdown. We should not push down it to Hive metastore when they are be using in partition predicates.

How was this patch tested?

Added a test case

@gatorsmile

This comment has been minimized.

Copy link
Member Author

commented Nov 7, 2017

@SparkQA

This comment has been minimized.

Copy link

commented Nov 7, 2017

Test build #83531 has finished for PR 19682 at commit be0e276.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.
*
* null-safe equality is not supported by Hive metastore partition predicate pushdown
*/
object OperatorsInMetastorePartitionFPD {

This comment has been minimized.

Copy link
@cloud-fan

cloud-fan Nov 7, 2017

Contributor

This name looks weird... maybe just SpecialBinaryComparison and explain the detail in the document.

This comment has been minimized.

Copy link
@gatorsmile

gatorsmile Nov 7, 2017

Author Member

Done.

@SparkQA

This comment has been minimized.

Copy link

commented Nov 7, 2017

Test build #83535 has finished for PR 19682 at commit be0e276.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@SparkQA

This comment has been minimized.

Copy link

commented Nov 7, 2017

Test build #83552 has finished for PR 19682 at commit bae7494.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@cloud-fan

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2017

thanks, merging to master!

@cloud-fan

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2017

can you send a new PR for 2.2? thanks!

@asfgit asfgit closed this in 0846a44 Nov 7, 2017

@mallman

This comment has been minimized.

Copy link
Contributor

commented Nov 8, 2017

Thanks for the fix!

gatorsmile added a commit to gatorsmile/spark that referenced this pull request Nov 11, 2017

[SPARK-22464][SQL] No pushdown for Hive metastore partition predicate…
…s containing null-safe equality

## What changes were proposed in this pull request?
`<=>` is not supported by Hive metastore partition predicate pushdown. We should not push down it to Hive metastore when they are be using in partition predicates.

## How was this patch tested?
Added a test case

Author: gatorsmile <gatorsmile@gmail.com>

Closes apache#19682 from gatorsmile/fixLimitPushDown.

# Conflicts:
#	sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
#	sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.