[GLUTEN-10236][VL] Support both sort and rss_sort shuffle writer for Celeborn#10244
[GLUTEN-10236][VL] Support both sort and rss_sort shuffle writer for Celeborn#10244kerwin-zk merged 7 commits intoapache:mainfrom
Conversation
github-actions
Bot
added
CORE
marin-ma
changed the title
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Cc @kerwin-zk |
|
Could you mention this in https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md#celeborn-support? There are two set of sort based shuffle implemented and supported in celeborn. Use the config to switch. How about Uniffle? Does it only support sort based shuffle, right? |
@FelixYBW The supported shuffle writer type for Uniffle is same as the local shuffle. We support both hash and row-based sort. We don't use the rss sort shuffle implementation for Uniffle. |
|
Run Gluten Clickhouse CI on x86 |
4 participants
By default, use rss sort shuffle implementation for Celeborn. User can set
spark.gluten.sql.columnar.shuffle.celeborn.useRssSort=falseto use row-based sort implementation.