Allow Nvidia driver script to set recommendations for LD_PRELOAD#754
Merged
bedroge merged 33 commits intoEESSI:2023.06-software.eessi.iofrom Jan 17, 2025
Merged
Conversation
|
Instance
|
|
Instance
|
|
Instance
|
Member
Author
|
Example output: [rocky@ip-172-31-27-81 software-layer]$ ./scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh --ld-preload --no-download
Found NVIDIA GPU driver version 545.23.08
Found host CUDA version 12.3
Using default list of libraries
Matched 48 CUDA Libraries
When attempting to use LD_PRELOAD we exclude anything related to graphics
libXext.so.6 is NOT in the provided preload list, filtering /lib64/libGL.so.1.
libXext.so.6 is NOT in the provided preload list, filtering /lib64/libGL.so.
libXext.so.6 is NOT in the provided preload list, filtering /lib64/libGLX_nvidia.so.0.
libXext.so.6 is NOT in the provided preload list, filtering /lib64/libGLX.so.0.
libXext.so.6 is NOT in the provided preload list, filtering /lib64/libGLX.so.
libwayland-server.so.0 is NOT in the provided preload list, filtering /lib64/libnvidia-egl-wayland.so.1.
libXext.so.6 is NOT in the provided preload list, filtering /lib64/libnvidia-fbc.so.1.
libXext.so.6 is NOT in the provided preload list, filtering /lib64/libnvidia-fbc.so.
libXNVCtrl.so.0 is NOT in the provided preload list, filtering /lib64/libnvidia-gtk3.so.545.23.08.
The recommended way to use LD_PRELOAD is to only use it when you need to:
export EESSI_GPU_LD_PRELOAD="/lib64/libcuda.so.1:/lib64/libcuda.so:/lib64/libcudadebugger.so.1:/lib64/libnvcuvid.so.1:/lib64/libnvcuvid.so:/lib64/libnvidia-cfg.so.1:/lib64/libnvidia-cfg.so:/lib64/libnvidia-eglcore.so.545.23.08:/lib64/libnvidia-encode.so.1:/lib64/libnvidia-encode.so:/lib64/libnvidia-glcore.so.545.23.08:/lib64/libnvidia-glsi.so.545.23.08:/lib64/libnvidia-glvkspirv.so.545.23.08:/lib64/libnvidia-gpucomp.so.545.23.08:/lib64/libnvidia-ml.so.1:/lib64/libnvidia-ml.so:/lib64/libnvidia-nvvm.so.4:/lib64/libnvidia-nvvm.so:/lib64/libnvidia-opencl.so.1:/lib64/libnvidia-opticalflow.so.1:/lib64/libnvidia-ptxjitcompiler.so.1:/lib64/libnvidia-ptxjitcompiler.so:/lib64/libnvidia-rtcore.so.545.23.08:/lib64/libnvidia-tls.so.545.23.08:/lib64/libnvoptix.so.1:/lib64/libOpenCL.so.1"
export EESSI_OVERRIDE_GPU_CHECK="1"
Then you can set LD_PRELOAD only when you want to run a GPU application, e.g.,
LD_PRELOAD="$EESSI_GPU_LD_PRELOAD" device_query |
boegel
added
2023.06-software.eessi.io
Contributor
|
@ocaisa There's duplicate entries here, |
ocaisa
commented
Oct 9, 2024
| # Filter out all symlinks and libraries that have missing library dependencies under EESSI | ||
| filtered_libraries=() | ||
| for library in "${matched_libraries[@]}"; do | ||
| if [ ! -L "$library" ]; then |
Member
Author
There was a problem hiding this comment.
This is too aggressive, instead we should just resolve the symlink and remove duplicate entries
Updates by the bot instance
|
Updates by the bot instance
|
Updates by the bot instance
|
|
New job on instance
|
ocaisa
changed the title
LD_PRELOAD
TopRichard
reviewed
Nov 5, 2024
Collaborator
|
Also tested the script within |
Accepted all except one Co-authored-by: TopRichard <121792457+TopRichard@users.noreply.github.com>
Member
Author
|
@TopRichard This will need to be re-tested now to make sure the changes haven't had an unintended impact |
Member
Author
|
bot: build repo:eessi.io-2023.06-software arch:x86_64/generic |
Updates by the bot instance
|
Updates by the bot instance
|
Updates by the bot instance
|
|
New job on instance
|
Collaborator
re-testing: Apptainer> /cvmfs/software.eessi.io/versions/2023.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh
Found host CUDA version 9.0
Found NVIDIA GPU driver version 535.129.03
Using downloaded list of libraries
Matched 41 CUDA Libraries
Successfully created symlink between latest and host in /cvmfs/software.eessi.io/host_injections/nvidia/aarch64
Successfully created symlink between /cvmfs/software.eessi.io/host_injections/nvidia/aarch64/latest and lib in /cvmfs/software.eessi.io/host_injections/2023.06/compat/linux/aarch64
Host NVIDIA GPU drivers linked successfully for EESSI |
TopRichard
added
bot:deploy
bedroge
added
the
bot:deploy
boegel
previously requested changes
Jan 17, 2025
Contributor
boegel
left a comment
boegel
left a comment
There was a problem hiding this comment.
I know this has just been deployed, so this review comes way too late, but I had a bunch of draft remarks that I somehow never submitted.
Take it as input for a follow-up PR...
|
|
||
| # This script links host libraries related to GPU drivers to a location where | ||
| # they can be found by the EESSI linker | ||
| # they can be found by the EESSI linker (or sets LD_PRELOAD as an |
Contributor
There was a problem hiding this comment.
It doesn't set $LD_PRELOAD, it prints how $EESSI_*LD_PRELOAD can be set
|
|
||
| get_nvlib_list() { | ||
| local nvliblist_url="https://raw.githubusercontent.com/apptainer/apptainer/main/etc/nvliblist.conf" | ||
| local default_nvlib_list=( |
Contributor
There was a problem hiding this comment.
Can we add a comment here mentioning how this list was compiled, and with which driver version this current list corresponds?
Member
Author
There was a problem hiding this comment.
The initial list basically comes from the location above, it's intended as a fall back
| "tls_test_.so" | ||
| ) | ||
|
|
||
| # Check if the function was called with the "default" argument |
Contributor
There was a problem hiding this comment.
nit-picking: please fix indent of the comment
|
|
||
| # Check if curl failed (i.e., the content is empty) | ||
| if [ -z "$nvliblist_content" ]; then | ||
| # Failed to download nvliblist.conf, using default list instead |
Contributor
There was a problem hiding this comment.
Should we print a warning here (via echo_yellow) that the download failed, and that we're doing a fallback?
|
|
||
| # Check if umask allows global read | ||
| if [ "$umask_octal" -gt 022 ]; then | ||
| fatal_error "The current umask ($current_umask) does not allow global read permissions, you'll want everyone to be able to read the created directory." |
Contributor
There was a problem hiding this comment.
Make a suggestion on how to fix this in the error message?
Also, can't we set $UMASK here to dance around this ourselves?
Member
Author
There was a problem hiding this comment.
Reluctant to silently override their $UMASK, but an option could be supported to allow this
| filtered_libraries=() | ||
| compat_filtered_libraries=() | ||
| for library in "${matched_libraries[@]}"; do | ||
| # Run ldd on the given binary and filter for "not found" libraries |
| compat_filtered_libraries=() | ||
| for library in "${matched_libraries[@]}"; do | ||
| # Run ldd on the given binary and filter for "not found" libraries | ||
| not_found_libs=$(ldd "$library" 2>/dev/null | grep "not found" | awk '{print $1}') |
Contributor
There was a problem hiding this comment.
We should avoid relying on awk, it may not be there, cut is way more likely to be there
| # Find the host ldconfig | ||
| host_ldconfig=$(get_host_ldconfig) | ||
| # Gather libraries on the host (_must_ be host ldconfig) | ||
| host_libraries=$($host_ldconfig -p | awk '{print $NF}') |
Contributor
There was a problem hiding this comment.
If we can, we should avoid relying on awk, it may not be installed, use cut instead?
| # Check if it is missing an so dep under EESSI | ||
| if [[ -z "$not_found_libs" ]]; then | ||
| # Resolve any symlink | ||
| realpath_library=$(realpath "$library") |
Contributor
There was a problem hiding this comment.
There's a small chance that realpath is not available, so perhaps we should check for this early and exit with an error if it's not available
| # Resolve any symlink | ||
| realpath_library=$(realpath "$library") | ||
| if [[ ! " ${filtered_libraries[@]} " =~ " $realpath_library " ]]; then | ||
| filtered_libraries+=("$realpath_library") |
Contributor
There was a problem hiding this comment.
line above assumes there's spaces?
Suggested change
| filtered_libraries+=("$realpath_library") | |
| filtered_libraries+=(" ${realpath_library} ") |
Contributor
|
@bedroge This was deployed, so PR should be merged too? |
Collaborator
Yes, the tarball has been ingested. |
bedroge
approved these changes
Jan 17, 2025
bedroge
dismissed
boegel’s stale review
This has already been deployed, requested changes can be done in a follow-up PR.
|
PR merged! Moved |
|
PR merged! Moved |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.