Bug in converter logic of ShuffleExchangeExec #1678
Description
Currently there are bugs in converter logic of ShuffleExchangeExec, there are duplicated cases for ShuffleExchangeExec in convertSparkPlan:
The corresponding test: AuronCheckConvertShuffleExchangeSuite seems didn't catch it. Actually when I run the test, it failed with following stacktrace:
None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at org.apache.auron.AuronCheckConvertShuffleExchangeSuite.$anonfun$new$2(AuronCheckConvertShuffleExchangeSuite.scala:65)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:306)
at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:304)
at org.apache.spark.sql.AuronQueryTest.withTable(AuronQueryTest.scala:29)
at org.apache.auron.AuronCheckConvertShuffleExchangeSuite.$anonfun$new$1(AuronCheckConvertShuffleExchangeSuite.scala:33)
I don't think these tests can pass because the plan contains AdaptiveSparkPlanExec:
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[c1#10], functions=[count(1)], output=[c1#10, count(1)#13L])
+- Exchange hashpartitioning(c1#10, 4), ENSURE_REQUIREMENTS, [plan_id=34]
+- HashAggregate(keys=[c1#10], functions=[partial_count(1)], output=[c1#10, count#18L])
+- Project [c1#10]
+- FileScan parquet default.test_shuffle[c1#10,part#12]
the function collectFirst here can not go through it:
val shuffleExchangeExec =
executePlan.queryExecution.executedPlan
.collectFirst { case shuffleExchangeExec: ShuffleExchangeExec =>
shuffleExchangeExec
}