Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LD_LIBRARY_PATH may not be well configured in Singurarity image #4

Open
ItokawaK opened this issue Jan 23, 2024 · 6 comments
Open

LD_LIBRARY_PATH may not be well configured in Singurarity image #4

ItokawaK opened this issue Jan 23, 2024 · 6 comments

Comments

@ItokawaK
Copy link

ItokawaK commented Jan 23, 2024

Hi,

Thank you for your interesting tool.

I got a following error when I executed herro inference in Singurarity.

$ singurarity run --nv herro.sif inference <args>
[00:01:02] Parsed 393084 reads.                                                                                                                                  
[00:00:16] Processing 1/? batch ⠠
[>---------------------------------------] 2/75403                                                                                                               
Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory
Aborted

So, I entered into the bash console first, configurered the LD_LIBRARY_PATH environmental variable inside, and executed herro inference as below.

$ singurarity exec --nv herro.sif bash
Singularity> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.7/compat/
Singularity> herro inference <args>

Then, it worked.

Best

@dominikstanojevic
Copy link
Member

Hello,

thank you for reporting the issue.

Unfortunately, I cannot reproduce the problem on the two test machines using a pre-build singularity image. Can you please specify if you're using a pre-build singularity image (or you built it from scratch) and a singularity version?

Best regrads,
Dominik

@ItokawaK
Copy link
Author

ItokawaK commented Jan 23, 2024

Hi Dominik,

I am using the pre-build image downloaded from aws S3.

Other specs are as follow.
OS: Ubuntu 20.04 LTS
singularity-ce: version 4.0.2

Thanks,
Kentaro

@dominikstanojevic
Copy link
Member

dominikstanojevic commented Jan 23, 2024

Hi Kentaro,

can you run echo $LD_LIBRARY_PATH after singurarity exec --nv herro.sif bash and paste the output?

Moreover, can you paste the output from ls -l /.singularity.d/libs? Thanks.

Best regards,
Dominik

@ItokawaK
Copy link
Author

Hi Dominik,

Thank you for looking into this issue.

$ singularity exec --nv ~/tools/herro/herro.sif bash
Singularity> echo $LD_LIBRARY_PATH 
/libs/libtorch/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
Singularity> ls -l /.singularity.d/libs/
-rw-r--r-- 1 root root     80512 Mar  8  2022 libEGL.so
-rw-r--r-- 1 root root     80512 Mar  8  2022 libEGL.so.1
-rw-r--r-- 1 root root   1333232 May 30  2023 libEGL_nvidia.so.0
-rw-r--r-- 1 root root    547152 Mar  8  2022 libGL.so
-rw-r--r-- 1 root root    547152 Mar  8  2022 libGL.so.1
-rw-r--r-- 1 root root     43336 Mar  8  2022 libGLESv1_CM.so
-rw-r--r-- 1 root root     43336 Mar  8  2022 libGLESv1_CM.so.1
-rw-r--r-- 1 root root     67880 May 30  2023 libGLESv1_CM_nvidia.so.1
-rw-r--r-- 1 root root     72008 Mar  8  2022 libGLESv2.so
-rw-r--r-- 1 root root     72008 Mar  8  2022 libGLESv2.so.2
-rw-r--r-- 1 root root    117032 May 30  2023 libGLESv2_nvidia.so.2
-rw-r--r-- 1 root root    141912 Mar  8  2022 libGLX.so
-rw-r--r-- 1 root root    141912 Mar  8  2022 libGLX.so.0
-rw-r--r-- 1 root root   1191352 May 30  2023 libGLX_nvidia.so.0
-rw-r--r-- 1 root root    715200 Mar  8  2022 libGLdispatch.so
-rw-r--r-- 1 root root    715200 Mar  8  2022 libGLdispatch.so.0
-rwxr-xr-x 1 root root     30856 Mar  9  2023 libOpenCL.so
-rwxr-xr-x 1 root root     30856 Mar  9  2023 libOpenCL.so.1
-rw-r--r-- 1 root root    174408 Mar  8  2022 libOpenGL.so
-rw-r--r-- 1 root root    174408 Mar  8  2022 libOpenGL.so.0
-rw-r--r-- 1 root root  29867944 May 30  2023 libcuda.so.1
-rw-r--r-- 1 root root   7781216 May 30  2023 libnvcuvid.so.1
-rw-r--r-- 1 root root    262512 May 30  2023 libnvidia-cfg.so.1
-rw-r--r-- 1 root root  56309912 May 30  2023 libnvidia-compiler.so.525.125.06
-rw-r--r-- 1 root root  42054840 May 30  2023 libnvidia-eglcore.so.525.125.06
-rw-r--r-- 1 root root    198720 May 30  2023 libnvidia-encode.so.1
-rw-r--r-- 1 root root    133816 May 30  2023 libnvidia-fbc.so.1
-rw-r--r-- 1 root root  44282616 May 30  2023 libnvidia-glcore.so.525.125.06
-rw-r--r-- 1 root root    652264 May 30  2023 libnvidia-glsi.so.525.125.06
-rw-r--r-- 1 root root  25056984 May 30  2023 libnvidia-glvkspirv.so.525.125.06
-rw-r--r-- 1 root root   1375912 Jan 19  2022 libnvidia-gtk2.so.470.57.01
-rw-r--r-- 1 root root   1384584 Jan 19  2022 libnvidia-gtk3.so.470.57.01
-rw-r--r-- 1 root root   1827448 May 30  2023 libnvidia-ml.so.1
-rw-r--r-- 1 root root  85957696 May 30  2023 libnvidia-nvvm.so.4
-rw-r--r-- 1 root root  22991312 May 30  2023 libnvidia-opencl.so.1
-rw-r--r-- 1 root root     67584 May 30  2023 libnvidia-opticalflow.so.1
-rw-r--r-- 1 root root  20727192 May 30  2023 libnvidia-ptxjitcompiler.so.1
-rw-r--r-- 1 root root 118080512 May 30  2023 libnvidia-rtcore.so.525.125.06
-rw-r--r-- 1 root root     18528 May 30  2023 libnvidia-tls.so.525.125.06
-rw-r--r-- 1 root root 201324280 May 30  2023 libnvoptix.so.1

Best,
Kentatro

@dominikstanojevic
Copy link
Member

Hi Kentaro,

I was checking on the third machine (with quite an old CUDA version) but still cannot reproduce the issue. Can you give me more info about your CUDA and NVIDIA driver versions?

Best regards,
Dominik

@ItokawaK
Copy link
Author

Hi Dominik,

I have only one GPU, NVIDIA RTX A6000.

nvidia-smi told

Driver Version: 525.125.06
CUDA Version: 12.0

I saw same result when I executed nvidia-smi in Singularity container.

Best,
Kentaro

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants