Skip to content

Do not convert into ShuffleExchangeExec into Native when shuffle manger not set #1372

@guixiaowen

Description

@guixiaowen

If the shuffle manager is not configured to use one of the Auron implementations (including AuronShuffleManager, AuronUniffleShuffleManager, or AuronCelebornShuffleManager), the ShuffleExchangeExec nodes will be converted into native execution plans.
During execution, the shuffle will then run through a non-Auron shuffle manager, which will trigger an exception.

Modify it to determine whether to convert to native based on the shuffle manager.

Eg:

set running conf

spark.auron.enable true
spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension

running sql

select c1, count(1) from test_shuffle group by c1

Exec Plan

`
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- == Current Plan ==
HashAggregate(keys=[a#1], functions=[count(1)], output=[a#1, count(1)#3L])
+- ShuffleQueryStage 0
+- NativeShuffleExchange hashpartitioning(a#1, 1000), ENSURE_REQUIREMENTS, [id=#29]
+- NativeHashAggregate HashAgg, [a#1], [partial_count(1)], [count#6L], 0
+- InputAdapter [#1]
+- NativeParquetScan xxx_us_xxx_parquet_datasource (FileScan parquet bigdata_qa.xxx_us_xxx_parquet_datasource[a#1] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[xxxxx/bigdata_qa/ns..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:string>)

`

Exception like

`
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (bigdata-s2102-hdp1187.nmg03.diditaxi.com executor 1): scala.MatchError: org.apache.spark.shuffle.sort.UnsafeShuffleWriter@c91b87 (of class org.apache.spark.shuffle.sort.UnsafeShuffleWriter)

`

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions