-
Notifications
You must be signed in to change notification settings - Fork 210
Description
If the shuffle manager is not configured to use one of the Auron implementations (including AuronShuffleManager, AuronUniffleShuffleManager, or AuronCelebornShuffleManager), the ShuffleExchangeExec nodes will be converted into native execution plans.
During execution, the shuffle will then run through a non-Auron shuffle manager, which will trigger an exception.
Modify it to determine whether to convert to native based on the shuffle manager.
Eg:
set running conf
spark.auron.enable true
spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension
running sql
select c1, count(1) from test_shuffle group by c1
Exec Plan
`
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- == Current Plan ==
HashAggregate(keys=[a#1], functions=[count(1)], output=[a#1, count(1)#3L])
+- ShuffleQueryStage 0
+- NativeShuffleExchange hashpartitioning(a#1, 1000), ENSURE_REQUIREMENTS, [id=#29]
+- NativeHashAggregate HashAgg, [a#1], [partial_count(1)], [count#6L], 0
+- InputAdapter [#1]
+- NativeParquetScan xxx_us_xxx_parquet_datasource (FileScan parquet bigdata_qa.xxx_us_xxx_parquet_datasource[a#1] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[xxxxx/bigdata_qa/ns..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:string>)
`
Exception like
`
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (bigdata-s2102-hdp1187.nmg03.diditaxi.com executor 1): scala.MatchError: org.apache.spark.shuffle.sort.UnsafeShuffleWriter@c91b87 (of class org.apache.spark.shuffle.sort.UnsafeShuffleWriter)
`