[SPARK-33108][BUILD] Remove sbt-dependency-graph SBT plugin

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to remove sbt-dependency-graph SBT plugin.

Why are the changes needed?

sbt-dependency-graph officially doesn't support SBT 1.3.x and it's broken due to NoSuchMethodError. This cannot be fixed in sbt-dependency-graph side at SBT 1.3.x

• https://github.com/sbt/sbt-dependency-graph

  Note: Under sbt >= 1.3.x some features might currently not work as expected or not at all (like dependencyLicenses).

$ build/sbt dependencyTree
Launching sbt from build/sbt-launch-1.3.13.jar
[info] welcome to sbt 1.3.13 (AdoptOpenJDK Java 1.8.0_252)
...
[error] java.lang.NoSuchMethodError: sbt.internal.LibraryManagement$.cachedUpdate(Lsbt/librarymanagement/DependencyResolution;Lsbt/librarymanagement/ModuleDescriptor;Lsbt/util/CacheStoreFactory;Ljava/lang/String;Lsbt/librarymanagement/UpdateConfiguration;Lscala/Function1;ZZZLsbt/librarymanagement/UnresolvedWarningConfiguration;Lsbt/librarymanagement/EvictionWarningOptions;ZLsbt/internal/librarymanagement/CompatibilityWarningOptions;Lsbt/util/Logger;)Lsbt/librarymanagement/UpdateReport;

ALTERNATIVES

• One alternative is coursier, but it requires coursier-based sbt launcher which is more intrusive.

  • https://get-coursier.io/docs/sbt-coursier.html#sbt-13x

    you'll have to use the coursier-based sbt launcher, via its custom sbt-extras launcher for example.

• Another alternative is moving to SBT 1.4.0 which uses sbt-dependency-graph as a built-in, but it's still new and will requires many change.

So, this PR aims to remove the broken plugin simply.

Does this PR introduce any user-facing change?

No. This is a dev-only change.

How was this patch tested?

Manual.

$ build/sbt dependencyTree
...
[error] Not a valid command: dependencyTree
[error] Not a valid project ID: dependencyTree
[error] Not a valid key: dependencyTree (similar: dependencyOverrides, sbtDependency, dependencyResolution)
[error] dependencyTree
[error]               ^

[dongjoon-hyun]

cc @srowen and @HyukjinKwon .
I found this while investigating SPARK-33104 .

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34214/

[srowen]

OK by me. It's the kind of plugin that really does not need to be defined in a project-specific build anyway.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34214/

[HyukjinKwon]

Okay to me too. Can we file a JIRA and leave a todo comment to recover?

[HyukjinKwon]

cc @gemelen and @ScrapCodes FYI

[dongjoon-hyun]

Thank you, @srowen and @HyukjinKwon . Sure. I'll add an IDed TODO comment.

Let's revisit this at SBT 1.4.

[dongjoon-hyun]

I'll merge this because this is not tested by any CI. I checked this manually.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34221/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34221/

[SparkQA]

Test build #129611 has finished for PR 29997 at commit 3895e0d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #129617 has finished for PR 29997 at commit 6cc043e.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.