[FEA] Detect multiple versions of the RAPIDS jar on the classpath at the same time

[revans2]

Is your feature request related to a problem? Please describe.
This is not going to work when the older jar is the first thing on the classpath, but at some point we can hopefully get to the point where the jars that don't have a check are not used so then we can reliably detect this.

It would really be nice if we had a way to detect that there were multiple versions of the plugin jar on the classpath. By default this should result in an error when we try to load, because the user might not have the version that they expect. We can have a config to allow it to keep running in this situation.

Describe the solution you'd like
The class loader can return an enumeration of multiple URLs for resources when there are duplicates on the classpath. We could try to look for cudf-java-version-info.properties, rapids4spark-version-info,.properties, and spark-rapids-jni-version-info.properties. If we find multiple of any of these on the classpath at a minimum we need to warn, and by default we probably should throw an exception.

We could even go as far as to read these files and see what is different. to include in the error message.

[revans2]

This just happened again to another customer. I filed NVIDIA/spark-rapids-tools#266 in tools to try and warn people about it, but at the same time if we lose a day working with a customer because of a simple mistake like this it really would be nice to just have the plugin crash to let you know something is wrong ahead of time. (possibly with a config to bypass it)

[mattahrens]

Scope: count number of Spark RAPIDS jars on the classpath; if number of Spark RAPIDS jars > 1, then fail with error message.

[gerashegalov]

here is an approach explored in the Scala shell that could be implemented somewhere around ShimLoader.

For example I have these rapids jars in my ivy cache

$ find ~/.ivy2/jars/ -name \*jar
/home/gshegalov/.ivy2/jars/com.nvidia_rapids-4-spark_2.12-23.08.1.jar
/home/gshegalov/.ivy2/jars/com.nvidia_rapids-4-spark_2.12-22.12.0.jar
/home/gshegalov/.ivy2/jars/com.nvidia_rapids-4-spark_2.12-23.10.0.jar
/home/gshegalov/.ivy2/jars/com.nvidia_rapids-4-spark_2.12-23.06.0.jar

~/dist/spark-3.3.0-bin-hadoop3/bin/spark-shell --jars $(find ~/.ivy2/jars/ -name '*rapids-4-spark_2.12*.jar' | xargs printf "%s,")

import scala.collection.JavaConverters._
val cl = classOf[com.nvidia.spark.rapids.SparkShimServiceProvider].getClassLoader
val rapidsJarURLs = cl.getResources("rapids4spark-version-info.properties").asScala.toList
scala> rapidsJarURLs.map { url => scala.io.Source.fromInputStream(url.openStream()).mkString("") }

yields

res5: List[String] =
List("version=23.08.1
cudf_version=23.08.0
user=
revision=ee2de89985ca0bc2b0b09eddd1efc3ccb9297d0b
branch=HEAD
date=2023-08-15T09:23:27Z
url=https://github.com/NVIDIA/spark-rapids.git
", "version=22.12.0
cudf_version=22.12.0
user=
revision=6bba270bc5ce68f009c0b3e2d5b41f562cf29abb
branch=HEAD
date=2022-12-12T18:58:25Z
url=https://github.com/NVIDIA/spark-rapids.git
", "version=23.10.0
cudf_version=23.10.0
user=
revision=33ca52bd21f7b5a91af1c8414f316155c3b0cc98
branch=HEAD
date=2023-10-24T09:28:08Z
url=https://github.com/NVIDIA/spark-rapids.git
", "version=23.06.0
cudf_version=23.06.0
user=
revision=643370df12d74b7de8717b2fb46432d94e523158
branch=HEAD
date=2023-06-19T14:10:29Z
url=https://github.com/NVIDIA/spark-rapids.git