[WIP][SPARK-31168][BUILD] Upgrade Scala to 2.12.12

[wangyum]

What changes were proposed in this pull request?

This PR(SPARK-31168) upgrade Scala from 2.12.10 to 2.12.12.

Scala 2.12.11 release notes:
Highlights

• Performance improvements in the collections library
• Performance improvements in the compiler
• Improvements to -Yrepl-class-based, an alternative internal REPL encoding that avoids deadlocks
• A new -Yrepl-use-magic-imports flag that avoids deep class nesting in the REPL, which can lead to deteriorating performance in long sessions
• Fix some toX methods that could expose the underlying mutability of a ListBuffer-generated collection

JDK 9+ support

• ASM was upgraded to 7.3.1, allowing the optimizer to run on JDK 13+
• :javap in the REPL now works on JDK 9+

Scala 2.12.12 release notes:
Highlights
This release adds compiler support for scala-async and backports substantial performance improvements to collections.

• The compiler now includes a phase that performs the transformation for the async/await DSL for working with Scala Futures.
  This transform is based on the macro implementation in scala-async but has been substantially rewritten to fix some
  long standing bugs, reduce the overhead of compilation and simplify the extension points for third party effect systems.
  scala-async 1.0.0 will be released shortly building atop this phase.
• immutable.{TreeSet, TreeMap} are now based on a backport of the 2.13.x implementation.
• The builders for immutable.{HashMap, HashSet} are more efficient -- they now use mutation during building to reduce memory churn.

Further highlights

• Minor improvements to compiler performance. The allocation rate of the compiler is reduced by 10%.

Why are the changes needed?

Improvement performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Exist test.

[dongjoon-hyun]

Thanks, @wangyum .
cc @srowen

[srowen]

I feel like there was some reason we couldn't update to 2.12.11; not sure if it's still an issue or not. Tests will help tell. Yes, generally fine otherwise to update.

[dongjoon-hyun]

The failure looks consistent. Could you take a look at that, @wangyum ?

[info] org.apache.spark.serializer.KryoSerializerSuite *** ABORTED *** (324 milliseconds)
[info]   java.lang.NoSuchFieldError: numNonEmptyBlocks
[info]   at org.apache.spark.scheduler.HighlyCompressedMapStatus.<init>(MapStatus.scala:174)

That might be another Scala bug.

[SparkQA]

Test build #125879 has finished for PR 29124 at commit 3adc82a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rednaxelafx]

Just a note: both -Yrepl-class-based and -Yrepl-use-magic-imports are very nice improvements to the Scala REPL, and I'd love to see them in Spark. But they can also potentially significantly impact how Spark's Scala REPL interact, so I wonder whether or not we need some more tests cases to show that the REPL behavior is consistent / has not regressed with regards to them. The test coverage in Spark's RPEL is quite weak...

[dongjoon-hyun]

Hi, @wangyum . Could you resolve the conflicts? Or, do you think we need to skip 2.12.12 like 2.12.11? If then, please update JIRA.

[dongjoon-hyun]

Given the current status, do we need to skip Scala 2.12.12 like 2.12.11, too?

[wangyum]

Yes. We should skip Scala 2.12.12. At least this bug should be fixed first: scala/bug#12096

[dongjoon-hyun]

Thank you so much for the investigation, @wangyum !