Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove spark.rapids.sql.nonUTC.enabled configuration option #10038

Merged
merged 4 commits into from
Dec 18, 2023
Merged

Remove spark.rapids.sql.nonUTC.enabled configuration option #10038

merged 4 commits into from
Dec 18, 2023

Conversation

winningsix
Copy link
Collaborator

This closes #9881.

Check framework already covers well for targeted expressions/execs. And we don't need extra configuration to turn on/off timezone related work. Also, it brings extra burden for adding more expression functions.

@winningsix
init commit
19a73d0 
Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com>
@winningsix winningsix marked this pull request as ready for review December 13, 2023 13:28
@winningsix
Copy link
Collaborator Author

build

@winningsix winningsix mentioned this pull request Dec 13, 2023
Merged
revans2
revans2 previously approved these changes Dec 13, 2023
View reviewed changes
@revans2
Copy link
Collaborator

revans2 commented Dec 13, 2023

Looks like the fallback tests for FromUTCTimestamp were relying on this config to pass.

[2023-12-13T14:38:53.308Z] FAILED ../../src/main/python/date_time_test.py::test_from_utc_timestamp_non_utc_fallback[Timestamp-Asia/Shanghai][DATAGEN_SEED=1702474979, ALLOW_NON_GPU(ProjectExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack.
[2023-12-13T14:38:53.308Z] FAILED ../../src/main/python/date_time_test.py::test_from_utc_timestamp_non_utc_fallback[Timestamp-EST][DATAGEN_SEED=1702474979, ALLOW_NON_GPU(ProjectExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack.
[2023-12-13T14:38:53.309Z] FAILED ../../src/main/python/date_time_test.py::test_from_utc_timestamp_non_utc_fallback[Timestamp-MST][DATAGEN_SEED=1702474979, ALLOW_NON_GPU(ProjectExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack.
[2023-12-13T14:38:53.309Z] FAILED ../../src/main/python/date_time_test.py::test_from_utc_timestamp_non_utc_fallback[Timestamp-VST][DATAGEN_SEED=1702474979, INJECT_OOM, ALLOW_NON_GPU(ProjectExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack.

We probably want to skip these tests if the timezone is not supported instead.

@winningsix
Copy link
Collaborator Author

build

@winningsix
Copy link
Collaborator Author

build

@winningsix winningsix merged commit 217d702 into NVIDIA:branch-24.02 Dec 18, 2023
37 of 38 checks passed
@winningsix winningsix deleted the remoteNonUTCConf branch December 18, 2023 01:51
@NVnavkumar NVnavkumar mentioned this pull request Dec 18, 2023
Merged
@sameerz sameerz added the task Work required that improves the product but is not user facing label Dec 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@res-life res-life res-life approved these changes

@revans2 revans2 revans2 left review comments

Assignees
No one assigned
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[TASK] Remove spark.rapids.sql.nonUTC.enabled configuration option
4 participants
@winningsix @revans2 @res-life @sameerz