Conflict between Julia's LLVM and system LLVM on FreeBSD

[ko56]

@iblis17
This problem manifests with Julia 1.1.1 on FreeBSD12.0, when trying to use OpenGL.

(v1.1) pkg> test GLFW
Testing GLFW
Status `/tmp/tmpalSg6q/Manifest.toml`
[9e28174c] BinDeps v0.8.10
[631607c0] CMake v1.1.2
...
[cf7118a7] UUIDs  [`@stdlib/UUIDs`]
[4ec0a83e] Unicode  [`@stdlib/Unicode`]
llibGL: OpenDriver: trying /usr/local/lib/dri/r600_dri.so
libGL: dlopen /usr/local/lib/dri/r600_dri.so failed (/usr/local/julia/julia11/lib/julia/libLLVM-6.0.so: version LLVM_6.0 required by /usr/local/lib/dri/r600_dri.so not found)
libGL error: unable to load driver: r600_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: r600
libGL: OpenDriver: trying /usr/local/lib/dri/swrast_dri.so
libGL: dlopen /usr/local/lib/dri/swrast_dri.so failed (/usr/local/julia/julia11/lib/julia/libLLVM-6.0.so: version LLVM_6.0 required by /usr/local/lib/dri/swrast_dri.so not found)
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
Testing GLFW tests passed 
(v1.1) pkg>

Now using ldd we see

$ ldd /usr/local/bin/julia11
/usr/local/bin/julia11:
   ...
    libLLVM-6.0.so => /usr/local/julia/julia11/lib/julia/libLLVM-6.0.so (0x801600000)
   ...

$ ldd /usr/local/lib/dri/r600_dri.so

/usr/local/lib/dri/r600_dri.so:
   ...
    libLLVM-6.0.so => /usr/local/llvm60/lib/libLLVM-6.0.so (0x801e00000)
   ... 
$

[iblislin]

An MWE on FreeBSD:

julia> using Libdl

julia> Libdl.dlopen("/usr/local/lib/dri/r600_dri.so")
ERROR: could not load library "/usr/local/lib/dri/r600_dri.so"
/data/usr/home/iblis/git/julia/usr/lib/libLLVM-6.0.so: version LLVM_6.0 required by /usr/local/lib/dri/r600_dri.so not found
Stacktrace:
...

But it works fine on Linux,

julia> Libdl.dlopen("/usr/lib/x86_64-linux-gnu/dri/r600_dri.so")
Ptr{Nothing} @0x00000000027adbc0

@ararslan I'm curious about how Linux handle that conflict. Any ideas?

[ararslan]

The only thing I could think of offhand would be that libraries are loaded in a different order or different places are searched first between Linux and FreeBSD. Not sure though, and I won't have time to investigate for a while. Maybe in the interim @staticfloat would have some ideas for debugging?

[iblislin]

I found this difference between Linux and FreeBSD.
In case of Linux, ldd reports that libjulia is not found.
Is that intended?

└─[iblis@Caneles]% uname -a
Linux Caneles 4.15.0-47-generic #50~16.04.1-Ubuntu SMP Fri Mar 15 16:06:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
└─[iblis@Caneles]% ldd ./bin/julia
        ...
        libjulia.so.1 => not found
        ...

Here is the output from FreeBSD.

└─[iblis@GaeBuidhe]% uname -a
FreeBSD GaeBuidhe 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  amd64
└─[iblis@GaeBuidhe]% ldd /home/iblis/bin/julia
/home/iblis/bin/julia:
        libjulia.so.1 => /data/usr/home/iblis/git/julia/usr/lib/libjulia.so.1 (0x800248000)
        ...

[ararslan]

Apparently @vchuravy has hit this before; the issue is not specific to FreeBSD. Perhaps he can share some insight here.

[vchuravy]

Well... This looks similar to JuliaGPU/OpenCL.jl#165 (comment) and #12644

#32462 (comment) on Linux one should be able to open both libraries and using symbol versioning to resolve to the right one.

[ararslan]

Okay, so speaking with Valentin in person, it seems it is a FreeBSD problem after all. FreeBSD names things a certain way that seems to collide with other symbol versioning.

[iblislin]

I met Hiroki (@hrs-allbsd), a FreeBSD Core Team member, at COSCUP 2019 in Taiwan yesterday.
I consulted him about this issue.

I quote the summary of the discussion from him:

First, the root cause was that two libraries with the same SONAME are loaded. Linux seems to load two symbol sets, but FreeBSD does not load the second library when it has the same SONAME as one which has been already loaded. So symbol versioning does not work in this case---as far as I can check some problem reports it seems that symbol versioning was introduced to solve the problem of system library vs julia private library. I personally think it is a wrong solution. A possible (safer) workaround is to rename the libLLVM-6.0.so in the private library directory in the building phase of lang/julia. libJL_LLVM-6.0.so should work.
I confirmed that a FreeBSD system with LLVM 8.0 does not suffer from this problem because it has libLLVM-8.0.so, not libLLVM-6.0.so in /usr/local/lib and /usr/local/lib/dri/r600_dri.so depends on libLLVM-8.0.so.

I will try to hack the LLVM build rules to change the SONAME first.
I also found a toggle USE_LLVM_SHLIB from doc, maybe this is another option.

Not sure what kind of side effects will be popped from these methods.

[vchuravy]

USE_LLVM_SHLIB is necessary for LLVM.jl renaming the shlib might work.

[iblislin]

Hiroki wrote a great patch for FreeBSD port lang/julia10:
https://people.allbsd.org/~hrs/FreeBSD/lang_julia10.20190822-1.diff

This patch redefines SONAME via setting LLVM_VERSION_MAJOR to 600,
It's an easy way to fix this issue in the ports tree.
llvm-config will return the modified name, so everything should work fine.

So, I think there might be two options for renaming it.

• libJLLLVM-6.0.so: I read the cmake scripts (but not deeply inspect) of LLVM, seems not configurable from outside at this moment.
• libLLVM-JL-6.0.so: I guess this can be achieve via abusing the LLVM_VERSION_MAJOR setting :p.