[VL] unsupported method

[FelixYBW]

Describe the bug
Spark compitable issue

To Reproduce
Vanilla Spark3.2.0.

[jinchengchenghh]

What's the compile command? Do you change the spark version to 3.2.0 in pom.xml?

[rui-mo]

Reproduced this error on Spark 3.2.0, but if we change the used spark version to 3.2.0 at pom.xml#L40, this error can be resolved.

23/03/02 18:43:44 WARN SparkShimProvider: Spark runtime version 3.2.0 is not matched with Gluten's fully tested version 3.2.2
java.lang.NoSuchMethodError: org.apache.spark.sql.execution.aggregate.HashAggregateExec.copy$default$2()Z
  at io.glutenproject.extension.columnar.StoreExpandGroupExpression$$anonfun$apply$1.applyOrElse(TransformHintRule.scala:113)
  at io.glutenproject.extension.columnar.StoreExpandGroupExpression$$anonfun$apply$1.applyOrElse(TransformHintRule.scala:110)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUpWithPruning$2(TreeNode.scala:535)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformUpWithPruning(TreeNode.scala:535)

[FelixYBW]

If we change to 3.2.0, can we pass 3.2.2?

[rui-mo]

If we change to 3.2.0, can we pass 3.2.2?

@FelixYBW No, if using Spark 3.2.0 in pom but submit the job with Spark 3.2.2, we got the similar error.

java.lang.NoSuchMethodError: org.apache.spark.sql.execution.aggregate.HashAggregateExec.copy$default$2()Lscala/collection/Seq;
  at io.glutenproject.extension.columnar.StoreExpandGroupExpression$$anonfun$apply$1.applyOrElse(TransformHintRule.scala:113)
  at io.glutenproject.extension.columnar.StoreExpandGroupExpression$$anonfun$apply$1.applyOrElse(TransformHintRule.scala:110)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUpWithPruning$2(TreeNode.scala:535)

[FelixYBW]

I see, @CodingCat how did you solve the issue?

[CodingCat]

mvn clean install -Pspark-3.2 -Pbackends-velox -DskipTests -Dspark32.version=3.2.0

and then submit in spark 3.2.0 environment, I think nowadays, it is harder to use different compile/run time versions of Spark even for maintenance releases

[FelixYBW]

Thank you. Let's manually fix the pom then.