[POC][DONOTMERGE] Spark Auto Upgrade

[holdenk]

This is a proof of concept only to see if the community is interested in something like this or not before I go too far down the development rabbit hole.

What changes were proposed in this pull request?

Some proof of concept tooling around adding support for better Spark upgrades.

Why are the changes needed?

API changes between 2.4 and 3.X

Does this PR introduce any user-facing change?

New tool

How was this patch tested?

Limited unit testing & manual run on spark-testing-base ( holdenk/spark-testing-base#330 )

[SparkQA]

Test build #135128 has finished for PR 31554 at commit 190a5fd.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class AccumulatorUpgrade extends SemanticRule(\"AccumulatorUpgrade\")
• class MigrateDeprecatedDataFrameReaderFuns extends SemanticRule(\"MigrateDeprecatedDataFrameReaderFuns\")
• class MigrateHiveContext extends SemanticRule(\"MigrateHiveContext\")
• class MigrateToSparkSessionBuilder extends SemanticRule(\"MigrateToSparkSessionBuilder\")
• class SparkAutoUpgrade extends SemanticRule(\"SparkAutoUpgrade\")
• case class Utils()(implicit val doc: SemanticDocument)
• class MagicMatcher(matchers: List[SymbolMatcher])
• class MagicMatcherLit[T <: meta.Lit](matchers: List[SymbolMatcher])

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39709/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39709/

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!