[SPARK-26793][CORE]Remove spark.shuffle.manager

[10110346]

What changes were proposed in this pull request?

Remove spark.shuffle.manager and update unit tests .
Currently, ShuffleManager always uses SortShuffleManager, so I think this configuration can be removed.

How was this patch tested?

Existing unit tests

[dongjoon-hyun]

cc @mridulm , @squito

[SparkQA]

Test build #101921 has finished for PR 23707 at commit 27a2a74.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

@10110346 wait, there's actually a bunch of ongoing work in this space that depends on this extensibility.

please start a discussion on dev@ before any further action. for reference, related work (one effort, there are others that plug-in to shuffle manager) https://issues.apache.org/jira/browse/SPARK-25299

[yuelimv]

@10110346 the option spark.shuffle.manager allows users to easily plug in other shuffle manager implementation, and I personally think it is a good option to preserve. As @felixcheung mentioned, we are currently having many on-going discussions and developments related to more reliable and flexible shuffle (for details, see SPARK-25299)

For instance, one effort made by us, named Splash, allows the use of other storage backends and network transports for exchanging shuffle data. Splash is configured via setting the spark.shuffle.manager option (see its README for more details).

[10110346]

OK ,thanks all, I will close it