External C Libraries and Runtime Dynamic Linking `dlopen` Not Picked Up by Enzyme

[halentin]

I was inspired by this post on the discourse, to try Enzyme through External Calls, and very impressed on how quickly i got some correct results.

One limitation of my use-case is, that i cant do load-time dynamic linking like in the example but i instead rely on runtime dynamic linking (via Libdl.dlopen and Libdl.dlsym). Right now only the load-time case works, as bitcode extraction looks for a string like jlptr_... which isnt present when loading via dlopen.

Here’s a minimal working example adapted from the Discourse post:

using Libdl
# works because library is linked at load-time
csquare(x::Cdouble) = ccall((:square, "./libsquare.so"), Cdouble, (Cdouble,), x)
@show first(first(autodiff(Reverse, csquare, Active, Active(3.14))))

# wrong derivative because bitcode isnt discovered from library linked at runtime
fptr = dlsym(dlopen("./libsquare.so"), :square)
csquare(x::Cdouble) = ccall(fptr, Cdouble, (Cdouble,), x)
@show first(first(autodiff(Reverse, csquare, Active, Active(3.14))))

Am I correct in assuming that this is not a fundamental limitation, but rather just a limitation of how Enzyme currently discovers bitcode for external functions?

Would it be feasible to extend Enzyme.jl to support this case? For example, by allowing users to pass the location of the .so containing bitcode manually (e.g. at dlopen time), and registering it for discovery.

I’d be happy to prototype a solution if this sounds like a reasonable direction. Some pointers on where to start would be appreciated.

[wsmoses]

yeah that's right (not a fundamental limitation, but we need to tell Enzyme how to find the bitcode). and absoutely please go for it! relevant code for the current importing is

Enzyme.jl/src/compiler/validation.jl

Line 503 in 13b0908

function try_import_llvmbc(mod::LLVM.Module, flib::String, fname::String, imported::Set{String})

and the callers of

[halentin]

I took another look at this, and figured out, that i can get around using Libdl and still load stuff dynamically at runtime by doing something this, which works with Enzyme.
@eval csquare(x::Float64) = ccall((:square, $path_to_binary), Float64, (Float64,), x)