Make it easier for embedded Julia to use a custom system image

[GunnarFarneback]

PackageCompiler does a great job at producing custom system images but deploying them for embedded targets is more complicated than it needs to be.

With the default system image, initializing embedded Julia is a simple matter of calling jl_init(). Under the hood this does dynamic loading introspection to find which library it is defined in (i.e. libjulia) and where this library lives on the filesystem, then deduces Julia's bindir from this information. After this it just calls jl_init_with_image with the bindir and the default relative location of the system image.

With a custom system image, the simplest way to initialize ought to be to call jl_init_with_image(NULL, system_image_path). However, the NULL first argument does something completely different than jl_init to determine bindir. Specifically it first checks the presence of the JULIA_BINDIR environment variable. If not present it uses uv_exepath to deduce bindir. The latter may be fine when initializing the julia binary, but for embedded julia it finds the path to the embedding executable, which typically lives somewhere else.

The result is that the embedding program needs to replicate the introspection done in jl_init itself, or have other ways of knowing where to find Julia's bindir. This is quite unnecessary since the code is already there in jl_init, just not available for this use case.

Suggestion:

• jl_init() is changed to an alias for jl_init_with_image(NULL, NULL).
• jl_init_with_image replaces the uv_exepath based lookup with the code currently employed in jl_init.

This has two potentially breaking consequences:

1. jl_init will honor the JULIA_BINDIR environment variable. This seems mostly like a positive change.
2. jl_init_with_image(NULL, ...) will depend on the location of libjulia instead of the location of the executable. Will this be an acceptable change? Specifically, will it work properly with the julia executable itself?

[staticfloat]

The bindir-locating code in jl_init() is also not correct in every case: although it is typical for the libraries to be located in /lib and for the binaries to be located in /bin, that may not be the case; in Debian packaging, for instance, it's typical to place libraries in /lib/x86_64-linux-gnu or something else, which would break that hardcoded pathing.

The alternative check, performed in jl_resolve_sysimg_location() (which is itself called from _julia_init())

I suggest we always do the _julia_init() method, and it is up to the embedding tool to set JULIA_BINDIR environment variable if it needs to.

[liskin]

@staticfloat Is there an issue for the faulty bindir-locating code or would you like me to create one? I was just hit by it on Debian, the embedding example fails:

$ LANG=C ./jul1 
ERROR: could not load library "/usr/lib/x86_64-linux-gnu/../bin/../lib/x86_64-linux-gnu/julia/sys.so"
/usr/lib/x86_64-linux-gnu/../bin/../lib/x86_64-linux-gnu/julia/sys.so: cannot open shared object file: No such file or directory

This also causes trouble for anyone who wishes to use Julia in R via JuliaCall: Non-Contradiction/JuliaCall#99

It can be worked around by hardcoding jl_init_with_image("/usr/bin", NULL) instead of jl_call(), but it'd be lovely if this just worked. If I understand you correctly, you're suggesting to give up and let the embedding tool set JULIA_BINDIR, but that would only move the problem to various libraries like JuliaCall which aren't any more likely to get it right.

Perhaps we can pass MULTIARCH_INSTALL from Makefile to jl_init and add another /../ if MULTIARCH_INSTALL is 1. What do you think?

[staticfloat]

There is a WIP branch that is attempting to solve a lot of the embedding problems, including the JULIA_BINDIR requirement, but it's not passing tests yet: #38160

[staticfloat]

Oh, I'm sorry, I misread this issue, I had two different issues mixed up in my head. :)

The loading story in v1.6 is changing somewhat (as evidenced by that PR) and in particular, we will have the option to be much more flexible in how we find libraries. In that PR, we explicitly find the location of the ${prefix}/lib/julia directory (henceforth called the "private libdir"), is that knowledge what is behind the issues with JULIA_BINDIR here?

[liskin]

@staticfloat Yes, I believe that explicit knowledge of where ${libdir}/julia is (libdir being either /usr/lib or /usr/lib/x86_64-linux-gnu or something like that) would definitely help in this case. And yeah, that dladdr(&get_libdir, …) seems nice. (But I'm definitely not an expert on shared libraries so I'm unable to say if this has any drawback or what.)

Thanks for your quick answer, btw. :-)

[mattcbro]

I'm keenly interested in this capability. Does anyone have an example of embedding julia in C but loading a custom system image? In particular what libraries do you have to replace/overwrite to get it to work?

If I can't get this sorted out they may just toss julia out of the solution space for our commercial project. We need to call julia functions from the java virtual machine.

[GunnarFarneback]

There is complete code for this in https://github.com/GunnarFarneback/DynamicallyLoadedEmbedding.jl.

[mattcbro]

@GunnarFarneback thanks for the link. It should help.