Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-36208][SQL] SparkScriptTransformation should support ANSI interval types #33419

Closed
wants to merge 2 commits into from
Closed

[SPARK-36208][SQL] SparkScriptTransformation should support ANSI interval types #33419

wants to merge 2 commits into from

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Jul 19, 2021

What changes were proposed in this pull request?

This PR changes BaseScriptTransformationExec for SparkScriptTransformationExec to support ANSI interval types.

Why are the changes needed?

SparkScriptTransformationExec support CalendarIntervalType so it's better to support ANSI interval types as well.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New test.

@github-actions github-actions bot added the SQL label Jul 19, 2021
@sarutak sarutak changed the title [SPARK-27790][SQL] SparkScriptTransformation should support ANSI interval types [SPARK-36208][SQL] SparkScriptTransformation should support ANSI interval types Jul 19, 2021
@SparkQA
Copy link

SparkQA commented Jul 19, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45769/

@SparkQA
Copy link

SparkQA commented Jul 19, 2021

Test build #141255 has finished for PR 33419 at commit f82ec65.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

MaxGekk
MaxGekk requested changes Jul 20, 2021
View reviewed changes
Copy link
Member

@MaxGekk MaxGekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, use utils functions.

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Outdated
Comment on lines 232 to 233
data => Duration.ofNanos(IntervalUtils.castStringToDTInterval(
UTF8String.fromString(data), start, end) * DateTimeConstants.NANOS_PER_MICROS),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
data => Duration.ofNanos(IntervalUtils.castStringToDTInterval(
UTF8String.fromString(data), start, end) * DateTimeConstants.NANOS_PER_MICROS),
data => IntervalUtils.microsToDuration(
IntervalUtils.castStringToDTInterval(UTF8String.fromString(data), start, end)),

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Outdated
@@ -223,6 +224,14 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
case CalendarIntervalType => wrapperConvertException(
data => IntervalUtils.stringToInterval(UTF8String.fromString(data)),
converter)
case YearMonthIntervalType(start, end) => wrapperConvertException(
data => Period.ofMonths(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
data => Period.ofMonths(
data => IntervalUtils.monthsToPeriod(

@sarutak
Copy link
Member Author

sarutak commented Jul 21, 2021

@MaxGekk Thanks, got it.

@SparkQA
Copy link

SparkQA commented Jul 21, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45894/

@SparkQA
Copy link

SparkQA commented Jul 21, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45894/

MaxGekk
MaxGekk approved these changes Jul 21, 2021
View reviewed changes
Copy link
Member

@MaxGekk MaxGekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for jenkins.

@SparkQA
Copy link

SparkQA commented Jul 21, 2021

Test build #141378 has finished for PR 33419 at commit b87cbf7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MaxGekk
Copy link
Member

MaxGekk commented Jul 21, 2021 •
edited

+1, LGTM. Merging to master.
Thank you, @sarutak .

@MaxGekk MaxGekk closed this in f56c7b7 Jul 21, 2021
MaxGekk pushed a commit to MaxGekk/spark that referenced this pull request Jul 21, 2021
@sarutak @MaxGekk
[SPARK-36208][SQL] SparkScriptTransformation should support ANSI inte…
665e72b 
…rval types

This PR changes `BaseScriptTransformationExec` for `SparkScriptTransformationExec` to support ANSI interval types.

`SparkScriptTransformationExec` support `CalendarIntervalType` so it's better to support ANSI interval types as well.

No.

New test.

Closes apache#33419 from sarutak/script-transformation-interval.

Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
(cherry picked from commit f56c7b7)
Signed-off-by: Max Gekk <max.gekk@gmail.com>
@MaxGekk
Copy link
Member

MaxGekk commented Jul 21, 2021

@sarutak Sorry, I haven't merged to branch-3.2. Here is the PR for it: #33463 @HyukjinKwon @gengliangwang Could you approve it, please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MaxGekk MaxGekk MaxGekk approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
3 participants
@sarutak @SparkQA @MaxGekk