Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-37026][ML][BUILD] Ensure the element type of ResolvedRFormula.terms is scala.Seq for Scala 2.13 #34301

Closed
wants to merge 1 commit into from

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Oct 17, 2021

What changes were proposed in this pull request?

This PR fixes the issue that scala.Seq[scala.collection.mutable.ArraySeq$ofRef] will be passed to ResolvedRFormula.terms though it expects scala.Seq[scala.Seq[String]] with Scala 2.13.
As of Scala 2.13, scala.Seq is scala.collection.immutable.Seq, so this issue happens.

Why are the changes needed?

Bug fix.
Due to this issue, ResolvedRFormula.toString throws ClassCastException.

java.lang.ClassCastException: scala.collection.mutable.ArraySeq$ofRef cannot be cast to scala.collection.immutable.Seq
        at scala.collection.immutable.List.map(List.scala:246)
        at scala.collection.immutable.List.map(List.scala:79)
        at org.apache.spark.ml.feature.ResolvedRFormula.toString(RFormulaParser.scala:143)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:748)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New test is added and build/sbt -Pscala-2.13 "testOnly org.apache.spark.ml.feature.RFormulaSuite" passes.
CIs should ensure that this change works with Scala 2.12 too.

@github-actions github-actions bot added the ML label Oct 17, 2021
@sarutak sarutak changed the title [SPARK-37026][ML] Ensure the element type of ResolvedRFormula.terms is scala.Seq for Scala 2.13 [SPARK-37026][ML][BUILD] Ensure the element type of ResolvedRFormula.terms is scala.Seq for Scala 2.13 Oct 17, 2021
@SparkQA
Copy link

SparkQA commented Oct 17, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48806/

@SparkQA
Copy link

SparkQA commented Oct 17, 2021

Test build #144327 has finished for PR 34301 at commit 76e499e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 17, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48806/

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants