Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-36920][SQL][FOLLOWUP] Fix input types of ABS(): numeric and ANSI intervals #34183

Closed
wants to merge 3 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Oct 5, 2021

What changes were proposed in this pull request?

Change allowed input types of Abs() from:

NumericType + CalendarIntervalType + YearMonthIntervalType + DayTimeIntervalType

to

NumericType + YearMonthIntervalType + DayTimeIntervalType

Why are the changes needed?

The changes make the error message more clear.

Before changes:

spark-sql> set spark.sql.legacy.interval.enabled=true;
spark.sql.legacy.interval.enabled	true
spark-sql> select abs(interval -10 days -20 minutes);
21/10/05 09:11:30 ERROR SparkSQLDriver: Failed in [select abs(interval -10 days -20 minutes)]
java.lang.ClassCastException: org.apache.spark.sql.types.CalendarIntervalType$ cannot be cast to org.apache.spark.sql.types.NumericType
	at org.apache.spark.sql.catalyst.util.TypeUtils$.getNumeric(TypeUtils.scala:77)
	at org.apache.spark.sql.catalyst.expressions.Abs.numeric$lzycompute(arithmetic.scala:172)
	at org.apache.spark.sql.catalyst.expressions.Abs.numeric(arithmetic.scala:169)

After:

spark.sql.legacy.interval.enabled	true
spark-sql> select abs(interval -10 days -20 minutes);
Error in query: cannot resolve 'abs(INTERVAL '-10 days -20 minutes')' due to data type mismatch: argument 1 requires (numeric or interval day to second or interval year to month) type, however, 'INTERVAL '-10 days -20 minutes'' is of interval type.; line 1 pos 7;
'Project [unresolvedalias(abs(-10 days -20 minutes, false), None)]
+- OneRowRelation

Does this PR introduce any user-facing change?

No, because the original changes of #34169 haven't released yet.

How was this patch tested?

Manually checked in the command line, see examples above.

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Test build #143841 has finished for PR 34183 at commit 56a29b5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

retest this please

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48354/

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48354/

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48355/

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48355/

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48360/

@MaxGekk
Copy link
Member Author

MaxGekk commented Oct 5, 2021

Merging to master.
@HyukjinKwon @sarutak @cloud-fan Thank you for review.

@MaxGekk MaxGekk closed this in 3ac0382 Oct 5, 2021
@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48360/

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Test build #143842 has finished for PR 34183 at commit 82b34ab.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 5, 2021

Test build #143847 has finished for PR 34183 at commit 070dda2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants