Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries #36753

Closed
wants to merge 3 commits into from

Commits on Jun 3, 2022

  1. Configuration menu
    Copy the full SHA
    ae808ea View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2022

  1. Fix source and binary incompatibilities

    JoshRosen authored and olaky committed Jun 7, 2022
    Configuration menu
    Copy the full SHA
    88f2e02 View commit details
    Browse the repository at this point in the history
  2. [SPARK-39259][SQL][TEST][FOLLOWUP] Fix Scala 2.13 `ClassCastException…

    …` in `ComputeCurrentTimeSuite`
    
    ### What changes were proposed in this pull request?
    
    Unfortunately, apache#36654 causes seven Scala 2.13 test failures in master/3.3 and Apache Spark 3.3 RC4.
    This PR aims to fix Scala 2.13 ClassCastException in the test code.
    
    ### Why are the changes needed?
    
    ```
    $ dev/change-scala-version.sh 2.13
    $ build/sbt "catalyst/testOnly *.ComputeCurrentTimeSuite" -Pscala-2.13
    ...
    [info] ComputeCurrentTimeSuite:
    [info] - analyzer should replace current_timestamp with literals *** FAILED *** (1 second, 189 milliseconds)
    [info]   java.lang.ClassCastException: scala.collection.mutable.ArrayBuffer cannot be cast to scala.collection.immutable.Seq
    [info]   at org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTimeSuite.literals(ComputeCurrentTimeSuite.scala:146)
    [info]   at org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTimeSuite.$anonfun$new$1(ComputeCurrentTimeSuite.scala:47)
    ...
    [info] *** 7 TESTS FAILED ***
    [error] Failed tests:
    [error] 	org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTimeSuite
    [error] (catalyst / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
    [error] Total time: 189 s (03:09), completed Jun 3, 2022 10:29:39 AM
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Pass the CIs and manually tests with Scala 2.13.
    
    ```
    $ dev/change-scala-version.sh 2.13
    $ build/sbt "catalyst/testOnly *.ComputeCurrentTimeSuite" -Pscala-2.13
    ...
    [info] ComputeCurrentTimeSuite:
    [info] - analyzer should replace current_timestamp with literals (545 milliseconds)
    [info] - analyzer should replace current_date with literals (11 milliseconds)
    [info] - SPARK-33469: Add current_timezone function (3 milliseconds)
    [info] - analyzer should replace localtimestamp with literals (4 milliseconds)
    [info] - analyzer should use equal timestamps across subqueries (182 milliseconds)
    [info] - analyzer should use consistent timestamps for different timezones (13 milliseconds)
    [info] - analyzer should use consistent timestamps for different timestamp functions (2 milliseconds)
    [info] Run completed in 1 second, 579 milliseconds.
    [info] Total number of tests run: 7
    [info] Suites: completed 1, aborted 0
    [info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
    [info] All tests passed.
    [success] Total time: 12 s, completed Jun 3, 2022, 10:54:03 AM
    ```
    
    Closes apache#36762 from dongjoon-hyun/SPARK-39259.
    
    Authored-by: Dongjoon Hyun <dongjoon@apache.org>
    Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
    dongjoon-hyun authored and olaky committed Jun 7, 2022
    Configuration menu
    Copy the full SHA
    7755ca6 View commit details
    Browse the repository at this point in the history