Error is thrown when multiple instances of the same es-hadoop library are deployed

[iliasbay]

I'm testing a tool that deploys and runs jobs in hadoop. It appears that the tool deploys libraries in 2 different ways, so they appear twice in a job's classpath.
When I added es-hadoop reference to that tool, I observed an error "Multiple ES-Hadoop versions detected in the classpath; please use only one", even though it was the same very version.
Is it possible to compare versions (or hash values) of es-hadoop in classpath and allow to proceed if they match?

[costin]

Classpath duplication is a problem. The check in fact has been added since way too many times people had issues because of it (old version vs new). And almost always it is hard to figure out the issue since folks are unaware of the classpath for so many reasons.
Further more, it is not unheard of folks deploying es-hadoop.jar along with es-hadoop-hive or es-hadoop-spark even though only one jar needs to be deployed.
Since adding the check, the bug reports on this front has decreased significantly.

I'm not sure what tool you are using but deploying the same jar twice is simply not a good idea.
I'd like to help however there are way too many reasons for the check to remain in place than to remove it.

[hindog]

FYI, I am running into this issue under some special circumstances (see below) and wonder if a compromise could be made, such as a sysprop that suppresses the check for advanced users. I can submit a PR if you're open to this approach, in which case I would need to know what name should I give the sysprop?

Right now I have to build a custom version of this library to disable that check to get our Spark jobs to run in our environment-- granted, we are running a somewhat esoteric setup. I'll spare most of the details, but the short of it is this:

We use an application jar with a manifest classpath that our job submitter will read and sync (as necessary) any missing/updated jars that it doesn't already contain in it's repository. After sync'ing needed jars, it submits a spark job with the same application jar and the --jars Spark-submit argument with the HDFS locations for all of the jars. Since Spark doesn't honor the application jar's manifest, it only sees one copy of the jar in it's classpath, but this library sees two (the one in the manifest [I think] and the one passed via --jars) and triggers the error.

[geekyouth]

https://stackoverflow.com/a/63353843/9633499