Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix allowMultipleJars recommend setting message #9922

Merged
merged 4 commits into from
Dec 5, 2023

Conversation

thirtiseven
Copy link
Collaborator

@thirtiseven thirtiseven commented Dec 1, 2023

Fixes #9897

This PR:

  • Make spark.rapids.sql.allowMultipleJars a public config.
  • Added a message when case of zero revision numbers on the classpath.
  • Do not recommend current config mode in error message.
  • Generated doc for this config.

Error message when set it to NEVER:

java.lang.IllegalArgumentException: requirement failed: Multiple rapids-4-spark jars found in the classpath:
revison: 399b1b5ac14687ca943e10ae8e09dec7f2433e45
	jar URL: jar:file:/home/haoyangl/spark-rapids/dist/target/rapids-4-spark_2.12-24.02.0-SNAPSHOT-cuda11.jar
	version=24.02.0-SNAPSHOT
	cudf_version=24.02.0-SNAPSHOT
	user=haoyangl
	revision=399b1b5ac14687ca943e10ae8e09dec7f2433e45
	branch=multipleJarMsg
	date=2023-12-04T08:44:09Z
	url=https://github.com/thirtiseven/spark-rapids.git
	jar URL: jar:file:/home/haoyangl/spark-rapids/dist/target/rapids-4-spark_2.12-24.02.0-SNAPSHOT-cuda112.jar
	version=24.02.0-SNAPSHOT
	cudf_version=24.02.0-SNAPSHOT
	user=haoyangl
	revision=399b1b5ac14687ca943e10ae8e09dec7f2433e45
	branch=multipleJarMsg
	date=2023-12-04T08:44:09Z
	url=https://github.com/thirtiseven/spark-rapids.git
Please make sure there is only one rapids-4-spark jar in the classpath. If it is impossible to fix the classpath you can suppress the error by setting spark.rapids.sql.allowMultipleJars to SAME_REVISION or ALWAYS. But setting it to ALWAYS can cause unpredictable behavior as the plugin may pick up the wrong jar.

SAME_REVISION:

23/12/04 16:51:29 WARN RapidsPluginUtils: Multiple rapids-4-spark jars found in the classpath:
revison: 399b1b5ac14687ca943e10ae8e09dec7f2433e45
	jar URL: jar:file:/home/haoyangl/spark-rapids/dist/target/rapids-4-spark_2.12-24.02.0-SNAPSHOT-cuda11.jar
	version=24.02.0-SNAPSHOT
	cudf_version=24.02.0-SNAPSHOT
	user=haoyangl
	revision=399b1b5ac14687ca943e10ae8e09dec7f2433e45
	branch=multipleJarMsg
	date=2023-12-04T08:44:09Z
	url=https://github.com/thirtiseven/spark-rapids.git
	jar URL: jar:file:/home/haoyangl/spark-rapids/dist/target/rapids-4-spark_2.12-24.02.0-SNAPSHOT-cuda112.jar
	version=24.02.0-SNAPSHOT
	cudf_version=24.02.0-SNAPSHOT
	user=haoyangl
	revision=399b1b5ac14687ca943e10ae8e09dec7f2433e45
	branch=multipleJarMsg
	date=2023-12-04T08:44:09Z
	url=https://github.com/thirtiseven/spark-rapids.git
Please make sure there is only one rapids-4-spark jar in the classpath. If it is impossible to fix the classpath you can suppress the error by setting spark.rapids.sql.allowMultipleJars to ALWAYS, but this can cause unpredictable behavior as the plugin may pick up the wrong jar.

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
@thirtiseven thirtiseven self-assigned this Dec 1, 2023
Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
@revans2
Copy link
Collaborator

revans2 commented Dec 4, 2023

build

@thirtiseven thirtiseven merged commit e83bbe3 into NVIDIA:branch-24.02 Dec 5, 2023
38 checks passed
@sameerz sameerz added the bug Something isn't working label Dec 9, 2023
@thirtiseven thirtiseven deleted the multipleJarMsg branch December 19, 2023 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Error message for multiple jars on classpath is wrong
3 participants