Skip to content

Conversation

@sunchao
Copy link
Member

@sunchao sunchao commented May 17, 2021

What changes were proposed in this pull request?

Fix test failure under Scala 2.13 by making test ScalaFunction StrLenMagic public.

Why are the changes needed?

A few tests are failing when using Scala 2.13 with error message like the following:

[info]   Cause: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 35, Column 121: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 35, Column 121: No a
pplicable constructor/method found for actual parameters "org.apache.spark.unsafe.types.UTF8String"; candidates are: "public int org.apache.spark.sql.connector.DataSourceV2FunctionSuite$StrLenMagic$.invoke(org.apache.spark.
unsafe.types.UTF8String)"
[info]   at org.apache.spark.sql.errors.QueryExecutionErrors$.compilerError(QueryExecutionErrors.scala:387)
[info]   at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:1415)
[info]   at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:1501)

This seems to be caused by the fact that the StrLenMagic is using private scope. After removing the private keyword the tests are now passing.

Does this PR introduce any user-facing change?

No

How was this patch tested?

$ dev/change-scala-version.sh 2.13
$ build/sbt "sql/testOnly *.DataSourceV2FunctionSuite" -Pscala-2.13

@github-actions github-actions bot added the SQL label May 17, 2021
@SparkQA
Copy link

SparkQA commented May 17, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43165/

@dongjoon-hyun
Copy link
Member

Thank you, @sunchao !

@dongjoon-hyun
Copy link
Member

I revised the PR description How was this patch tested? section because this recovers Scala 2.13 tests.

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. I manually verified with Scala 2.13. Merged to master.

$ dev/change-scala-version.sh 2.13
$ build/sbt "sql/testOnly *.DataSourceV2FunctionSuite" -Pscala-2.13
...
[info] Run completed in 9 seconds, 581 milliseconds.
[info] Total number of tests run: 29
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 29, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 234 s (03:54), completed May 17, 2021 4:15:35 PM

@sunchao
Copy link
Member Author

sunchao commented May 17, 2021

Thanks @dongjoon-hyun !

@SparkQA
Copy link

SparkQA commented May 18, 2021

Test build #138645 has finished for PR 32575 at commit 4eb00fe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@sunchao sunchao deleted the SPARK-34981-follow-up branch May 18, 2021 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants