dlopen() and DL_LOAD_PATH doesn't properly load secondary dependencies

[swadey]

Trying to load libhdf5.so which depends on libsz.so. If I set DL_LOAD_PATH it fails to load libsz.so even though it's in the same directory (see details here: JuliaIO/HDF5.jl#97)

[JeffBezanson]

I don't think there's anything we can do about this; DL_LOAD_PATH can only affect where we look for libhdf5.so. The linux dynamic linker will not look in the current directory or in the same directory as the dlopen'ed library. My guess is setting LD_LIBRARY_PATH is the only option. Out of curiosity, what is the output of ldd libhdf5.so?

[swadey]

I figured as much. It looks like dlopen() tries to load a file per path in DL_LOAD_PATH per extension. Perhaps a solution could be to just adjust the environment using DL_LOAD_PATH (i.e. temporarily append this variable on the LD_LIBRARY_PATH for the duration of dlopen())?

Here's the output ldd without LD_LIBRARY_PATH set:

[swade@mt9 lib]$ ldd libhdf5.so 
        linux-vdso.so.1 =>  (0x00007fffbcdfe000)
        libsz.so.2 => not found
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc3adf06000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc3adcfd000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc3adaf9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc3ad7fd000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc3ad43c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc3ad21f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc3ae5f3000)

and with:

[swade@mt9 lib]$ env LD_LIBRARY_PATH=. ldd libhdf5.so                                                                                                                                 
        linux-vdso.so.1 =>  (0x00007f4690ee5000)
        libsz.so.2 => ./libsz.so.2 (0x00007f4690918000)
        libz.so.1 => ./libz.so.1 (0x00007f4690700000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f46904db000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f46902d7000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f468ffdb000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f468fc1a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f468f9fd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4690ee7000)

[JeffBezanson]

It would be nice for it to "just work", but modifying the LD_LIBRARY_PATH behind your back feels really sketchy to me.

[swadey]

Yeah I agree. It does.

thanks,
wade

On May 28, 2014, at 10:56 AM, Jeff Bezanson notifications@github.com wrote:

It would be nice for it to "just work", but modifying the LD_LIBRARY_PATH behind your back feels really sketchy to me.

—
Reply to this email directly or view it on GitHub.

[vtjnash]

the linker only reads the LD_LIBRARY_PATH environment variable once, at startup. later changes are ignored. you can't change the load path programmatically, AFAIK, or I would have used that API instead of creating DL_LOAD_PATH

[vtjnash]

I think that the linker will use libraries you have already opened. So, in theory, you could parse the library headers and load all of the libraries AOT

[staticfloat]

I think that the linker will use libraries you have already opened.

Yes, agreed here. At least, I believe that happens on Linux and Mac, not sure about windows.

[vtjnash]

Also windows. Windows also looks in the currently directory, and let's you change the lookup PATH at runtime

I think it may get more complicated when using absolute paths, however

[BobPortmann]

@vtjnash said above:

I think that the linker will use libraries you have already opened. So, in theory, you could parse the library headers and load all of the libraries AOT

I don't think this works (but I'm likely doing something wrong):

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin13.4.0

julia> push!(Libdl.DL_LOAD_PATH, "/Applications/exelis/idl85/bin/bin.darwin.x86_64")
3-element Array{String,1}:
 "@loader_path/julia"                              
 "@loader_path"                                    
 "/Applications/exelis/idl85/bin/bin.darwin.x86_64"

julia> Libdl.dlopen("libMesaGL6_2.dylib")
Ptr{Void} @0x00007feed4a03620

julia> Libdl.dlpath("libMesaGL6_2")
"/Applications/exelis/idl85/bin/bin.darwin.x86_64/libMesaGL6_2.dylib"

julia> Libdl.dlpath("libMesaGL6_2") in Libdl.dllist()
true

julia> Libdl.dlopen("libMesaGLU6_2.dylib")
ERROR: could not load library "libMesaGLU6_2.dylib"
dlopen(/Applications/exelis/idl85/bin/bin.darwin.x86_64/libMesaGLU6_2.dylib, 1): Library not loaded: libMesaGL6_2.dylib
  Referenced from: /Applications/exelis/idl85/bin/bin.darwin.x86_64/libMesaGLU6_2.dylib
  Reason: image not found
 in dlopen(::String, ::UInt32) at ./libdl.jl:90 (repeats 2 times)
 in dlopen(::String) at /Users/portmann/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?

Notice that the library that "could not load" is in the dllist? Is there a way to get this library to load without using DYLD_FALLBACK_LIBRARY_PATH?

[vtjnash]

Every platform is different. On macOS, sufficient information to back out what can go wrong is available from doing otool -L /Applications/exelis/idl85/bin/bin.darwin.x86_64/libMesaGLU6_2.dylib, although sometimes the -l flag is needed instead. Generally, macOS is the hardest and easiest platform to work with: it provides the most rope, but what you do with it is up to the person who compiled the dylib.

[BobPortmann]

Yes, I am learning things I don't want to know :-) In any case, I've found a workaround. The confusing thing to me is that this does not work (note all code is run in 0.5.0, see header above):

julia> cd("/Applications/exelis/idl85/bin/bin.darwin.x86_64") do
           ccall((:IDL_Init, "/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl"), Cint, 
                 (Cint, Ptr{Cint}, Ptr{Ptr{UInt8}}),0, C_NULL, C_NULL)
       end
ERROR: error compiling cd: could not load library "/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl"
dlopen(/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib, 1): Library not loaded: libMesaGL6_2.dylib
  Referenced from: /Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib
  Reason: image not found

or this (note I just removed the seemingly unnecessary full path from the library name, note that the error changed)

julia> cd("/Applications/exelis/idl85/bin/bin.darwin.x86_64") do
           ccall((:IDL_Init, "libidl"), Cint, 
                 (Cint, Ptr{Cint}, Ptr{Ptr{UInt8}}),0, C_NULL, C_NULL)
       end
ERROR: error compiling cd: could not load library "libidl"
dlopen(libidl.dylib, 1): image not found

but this does

julia> cd("/Applications/exelis/idl85/bin/bin.darwin.x86_64") do
           Libdl.dlopen("libidl.dylib")
       end
Ptr{Void} @0x00007fa80ac05560

julia> ccall((:IDL_Init, "/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl"), Cint, 
             (Cint, Ptr{Cint}, Ptr{Ptr{UInt8}}),0, C_NULL, C_NULL)
IDL Version 8.5, Mac OS X (darwin x86_64 m64).
(c) 2015, Exelis Visual Information Solutions, Inc., a subsidiary of Harris Corporation.

The ccall cannot be in the cd but dlopen can be? But it looks like I have a way forward now.

[vtjnash]

As I hinted above, I believe this is due to an error in the command line originally used to build libidl. The output of otool -L would let you see if the library was originally compiled to be named ./libMesaGL6_2.dylib rather than a correct name like @loader_path/libMesaGL6_2.dylib.

[BobPortmann]

Yes, it only has a local path.

> otool -L /Applications/exelis/idl85/bin/bin.darwin.x86_64/libMesaGLU6_2.dylib
/Applications/exelis/idl85/bin/bin.darwin.x86_64/libMesaGLU6_2.dylib:
        libMesaGLU6_2.dylib (compatibility version 0.0.0, current version 0.0.0)
        libMesaGL6_2.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

and libidl has several

> otool -L /Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib 
/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib:
        libidl.8.5.dylib (compatibility version 0.0.0, current version 0.0.0)
        /opt/X11/lib/libfreetype.6.dylib (compatibility version 15.0.0, current version 15.1.0)
        /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 57.0.0)
        /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 19.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.0.0)
        libMesaGLU6_2.dylib (compatibility version 0.0.0, current version 0.0.0)
        libMesaGL6_2.dylib (compatibility version 0.0.0, current version 0.0.0)
        libOSMesa6_2.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
        /opt/X11/lib/libXp.6.dylib (compatibility version 9.0.0, current version 9.0.0)
        libXm.3.0.2.dylib (compatibility version 4.0.0, current version 4.2.0)
        /opt/X11/lib/libXpm.4.dylib (compatibility version 16.0.0, current version 16.0.0)
        /opt/X11/lib/libXmu.6.dylib (compatibility version 9.0.0, current version 9.0.0)
        /opt/X11/lib/libXext.6.dylib (compatibility version 11.0.0, current version 11.0.0)
        /opt/X11/lib/libXt.6.dylib (compatibility version 7.0.0, current version 7.0.0)
        /opt/X11/lib/libXinerama.1.dylib (compatibility version 2.0.0, current version 2.0.0)
        /opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
        /opt/X11/lib/libICE.6.dylib (compatibility version 10.0.0, current version 10.0.0)
        /opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current version 10.0.0)
        /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1187.0.0)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

It is commercial software but I can put in a request for them to fix it. At least there is a workaround.