[FEA] create the plugin package capable of storing conflicting multiple versions of same named classes

[gerashegalov]

Is your feature request related to a problem? Please describe.
This feature contributes to #3223. Spark does not provide link compatibility. Therefore identical classes compiled against different spark versions may result in incompatible bytecode. We need to save multiple copies of such classes in the plugin jar. Standard way of using ASM-based shading/relocation does not work well with Scala.

Describe the solution you'd like

This issue proposes an approach similar to ParallelWorldClassLoader. Instead of overriding findClass in Spark we can lean on MutableURLClassLoader in conjunction with JarURLConnection JAR URLs

The package will consist of three types of areas

1. a few publicly documented classes in the conventional layout
2. a large fraction of classes whose bytecode is identical under all supported Spark versions
3. a smaller fraction of classes that differ under one of the supported Spark vesions

$ jar tvf rapids-4-spark_2.12.jar
com/nvidia/spark/SQLPlugin.class
spark3xx-common/com/nvidia/spark/rapids/CastExprMeta.class
spark301/org/apache/spark/sql/rapids/GpuUnaryMinus.class    
spark311/org/apache/spark/sql/rapids/GpuUnaryMinus.class
spark320/org/apache/spark/sql/rapids/GpuUnaryMinus.class

So each shim can see a consistent parallel world without conflicts by referencing only one conflicting directory.

E.g., Spark 3.2.0 Shim will use

jar:file:/home/spark/rapids-4-spark_2.12-21.10.jar!/spark3xx-common/
jar:file:/home/spark/rapids-4-spark_2.12-21.10.jar!/spark320/

Spark 3.1.1 will use

jar:file:/home/spark/rapids-4-spark_2.12-21.10.jar!/spark3xx-common/
jar:file:/home/spark/rapids-4-spark_2.12-21.10.jar!/spark311/

Describe alternatives you've considered

• shading
• JDK ParallelWorldClassLoader
• replicating copies of the same under different package names.

Additional context

• this issue should be broken down in packaging and actual class loader implementation

[jlowe]

@tgravescs this is now complete after #3411, correct?

[gerashegalov]

Completed via #3411