Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] test_cast_date_integral fails on databricks 3.4.1 #9977

Closed
revans2 opened this issue Dec 6, 2023 · 3 comments · Fixed by #9984
Closed

[BUG] test_cast_date_integral fails on databricks 3.4.1 #9977

revans2 opened this issue Dec 6, 2023 · 3 comments · Fixed by #9984
Assignees
Labels
bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Dec 6, 2023

Describe the bug
#9957 added a new test that fails on databricks because not all of the casts are supported on databricks by default.

[2023-12-06T14:15:54.218Z] 
[2023-12-06T14:15:54.218Z] [gw2]�[36m [ 12%] �[0m�[32mPASSED�[0m ../../src/main/python/cast_test.py::test_cast_fallback_not_UTC[SetValues(not_null)(StringType(),['2023-03-20 10:38:50', '2023-03-20 10:39:02'])-TIMESTAMP][DATAGEN_SEED=1701870720, INJECT_OOM, ALLOW_NON_GPU(ProjectExec)] 
[2023-12-06T14:15:54.471Z] ../../src/main/python/cast_test.py::test_cast_date_integral[DATAGEN_SEED=1701870720] 23/12/06 14:15:54 WARN GpuOverrides: 
[2023-12-06T14:15:54.472Z]   ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
[2023-12-06T14:15:54.472Z]     @Expression <AttributeReference> a#25534 could run on GPU
[2023-12-06T14:15:54.472Z]     @Expression <AttributeReference> b#25535 could run on GPU
[2023-12-06T14:15:54.472Z] 
[2023-12-06T14:15:54.725Z] 23/12/06 14:15:54 WARN GpuOverrides: 
[2023-12-06T14:15:54.726Z] !Exec <ProjectExec> cannot run on GPU because not all expressions can be replaced
[2023-12-06T14:15:54.726Z]   @Expression <Alias> cast(a#102050 as tinyint) AS a#102052 could run on GPU
[2023-12-06T14:15:54.726Z]     !Expression <Cast> cast(a#102050 as tinyint) cannot run on GPU because Cast from DateType to ByteType is not supported
[2023-12-06T14:15:54.726Z]       @Expression <AttributeReference> a#102050 could run on GPU
[2023-12-06T14:15:54.726Z]   @Expression <Alias> cast(a#102050 as smallint) AS a#102053 could run on GPU
[2023-12-06T14:15:54.726Z]     !Expression <Cast> cast(a#102050 as smallint) cannot run on GPU because Cast from DateType to ShortType is not supported
[2023-12-06T14:15:54.727Z]       @Expression <AttributeReference> a#102050 could run on GPU
[2023-12-06T14:15:54.727Z]   @Expression <Alias> cast(a#102050 as int) AS a#102054 could run on GPU
[2023-12-06T14:15:54.727Z]     !Expression <Cast> cast(a#102050 as int) cannot run on GPU because Cast from DateType to IntegerType is not supported
[2023-12-06T14:15:54.727Z]       @Expression <AttributeReference> a#102050 could run on GPU
[2023-12-06T14:15:54.727Z]   @Expression <Alias> cast(a#102050 as bigint) AS a#102055L could run on GPU
[2023-12-06T14:15:54.727Z]     !Expression <Cast> cast(a#102050 as bigint) cannot run on GPU because Cast from DateType to LongType is not supported
[2023-12-06T14:15:54.727Z]       @Expression <AttributeReference> a#102050 could run on GPU
[2023-12-06T14:15:54.727Z]   ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
[2023-12-06T14:15:54.727Z]     @Expression <AttributeReference> a#102050 could run on GPU
[2023-12-06T14:15:54.727Z] ```

Not sure if this is something special that was pulled back into databricks to make it more closely match ANSI SQL or what.
@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Dec 6, 2023
@NvTimLiu
Copy link
Collaborator

NvTimLiu commented Dec 7, 2023

Besides Databricks environment, the same failure also occurred on our nightly spark-3.4.1 & spark-3.5.0 integration tests.

@NVnavkumar
Copy link
Collaborator

This failure also occurs in Spark 3.4.0 as well

@res-life
Copy link
Collaborator

res-life commented Dec 7, 2023

Fix: #9984

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
5 participants