Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-36022][SQL] Respect interval fields in extract #33247

Closed
wants to merge 2 commits into from

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Jul 7, 2021

What changes were proposed in this pull request?

This PR fixes an issue about extract.
Extract should process only existing fields of interval types. For example:

spark-sql> SELECT EXTRACT(MONTH FROM INTERVAL '2021-11' YEAR TO MONTH);
11
spark-sql> SELECT EXTRACT(MONTH FROM INTERVAL '2021' YEAR);
0

The last command should fail as the month field doesn't present in INTERVAL YEAR.

Why are the changes needed?

Bug fix.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New tests.

@github-actions github-actions bot added the SQL label Jul 7, 2021
@SparkQA
Copy link

SparkQA commented Jul 7, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45268/

@SparkQA
Copy link

SparkQA commented Jul 7, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45268/

@SparkQA
Copy link

SparkQA commented Jul 7, 2021

Test build #140757 has finished for PR 33247 at commit 6524f02.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MaxGekk
Copy link
Member

MaxGekk commented Jul 7, 2021

@sarutak The failure in GA is related to the changes, it seems:

[info] - extract.sql *** FAILED *** (2 seconds, 541 milliseconds)
[info]   extract.sql
[info]   Expected "struct<[extract(SECOND FROM INTERVAL '123 12:34:56.789123' DAY TO SECOND):decimal(8,6)]>", but got "struct<[]>" Schema did not match for query #119
[info]   select extract(SECOND from interval '123 12:34:56.789123123' DAY TO SECOND): -- !query
[info]   select extract(SECOND from interval '123 12:34:56.789123123' DAY TO SECOND)
[info]   -- !query schema
[info]   struct<>
[info]   -- !query output
[info]   java.lang.ClassCastException

Could you fix it.

@sarutak
Copy link
Member Author

sarutak commented Jul 8, 2021

@MaxGekk Thank you for letting me know. I'll look into it.

@SparkQA
Copy link

SparkQA commented Jul 8, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45282/

@SparkQA
Copy link

SparkQA commented Jul 8, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45282/

@SparkQA
Copy link

SparkQA commented Jul 8, 2021

Test build #140770 has finished for PR 33247 at commit c4ff41b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@MaxGekk MaxGekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Merging to master/3.2.
Thank you, @sarutak .

@MaxGekk MaxGekk closed this in 39002cb Jul 8, 2021
MaxGekk pushed a commit that referenced this pull request Jul 8, 2021
### What changes were proposed in this pull request?

This PR fixes an issue about `extract`.
`Extract` should process only existing fields of interval types. For example:

```
spark-sql> SELECT EXTRACT(MONTH FROM INTERVAL '2021-11' YEAR TO MONTH);
11
spark-sql> SELECT EXTRACT(MONTH FROM INTERVAL '2021' YEAR);
0
```
The last command should fail as the month field doesn't present in INTERVAL YEAR.

### Why are the changes needed?

Bug fix.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

New tests.

Closes #33247 from sarutak/fix-extract-interval.

Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
(cherry picked from commit 39002cb)
Signed-off-by: Max Gekk <max.gekk@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants