update the default spark-rapids version to 2512#1370
update the default spark-rapids version to 2512#1370jayadeep-jayaraman merged 1 commit intoGoogleCloudDataproc:mainfrom
Conversation
Signed-off-by: liyuan <yuali@nvidia.com>
Summary of ChangesHello @nvliyuan, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request performs a routine update of the default Spark-RAPIDS version used within the project. The change ensures that the system leverages the most recent Spark-RAPIDS plugin release, incorporating any new features, performance improvements, or bug fixes that come with version 25.12.0. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the default Spark-RAPIDS version to 25.12.0. While the version update itself is correct, it highlights an existing issue where the script uses a hardcoded repository URL that does not contain this new version, which will lead to a download failure. A specific comment has been added to address this critical issue. Additionally, I've noticed significant code duplication between this script and gpu/install_gpu_driver.sh. Consolidating the logic could improve maintainability and prevent such inconsistencies in the future.
|
|
||
| # Update SPARK RAPIDS config | ||
| readonly DEFAULT_SPARK_RAPIDS_VERSION="25.10.0" | ||
| readonly DEFAULT_SPARK_RAPIDS_VERSION="25.12.0" |
There was a problem hiding this comment.
With this version update, the script will fail to download the Spark-RAPIDS JAR. The install_spark_rapids function uses a hardcoded nvidia_repo_url (https://repo1.maven.org/maven2/com/nvidia) which does not host versions newer than 24.02.0.
Newer versions, including 25.12.0, are available at a different repository. The gpu/install_gpu_driver.sh script contains logic to switch to https://edge.urm.nvidia.com/artifactory/sw-spark-maven/com/nvidia for newer Dataproc image versions.
To fix this, the install_spark_rapids function needs to be updated to use the correct repository URL for this new version. It would be best to adopt the dynamic URL selection logic present in gpu/install_gpu_driver.sh to make this script more robust for future version updates.
|
@cjac @jayadeep-jayaraman can you please help check? thx |
|
/gcbrun |
3 participants
Update the spark-rapids jar version after plugin release