Support for using a custom system image?

[jlapeyre]

pyjulia allows starting julia with a system image, like this Julia(sysimage=sys_image_path). Is something like this possible in juliacall?

I've started a project with pyjulia. I include a subdirectory and Julia code to compile a system image with PackageCompiler. I include a python function that performs all the steps, instantiating the compilation project and doing the compilation. Then, when my python module is imported the next time and the custom system image is found, it is used. It would be great to be able to do something like this with pythoncall.

[cjdoris]

It would be easy enough to add an environment variable to select a sysimage to use. I'll put it on the to-do list

I think what you want though is for a package to be able to choose the sysimage? The problem with that is that the sysimage is a global choice, so what happens if another package also wants to choose a sysimage? Something possible could be to have a mechanism to automatically create sysimages, and packages could define scripts to get run when creating the image.

[jlapeyre]

Exactly. I want the user to install my python package, and then be able to push a button to make the code load and run more like statically compiled code. So this really requires a per-package system image. Having a per-package Julia installation might be overkill. Although I can imagine two projects that use pyjulia (or juliacall) might require or perform optimally with differing versions of Julia.

I'm not super familiar with culture of python packages. In particular, is the installed package tree meant to be immutable, as it is in Julia? But, at the moment, I store this system image and some user choices in the source tree of the Python package.

What I have is a bit fragile. Here is the directory where the sysimage is created. And for simplicity, it is not installed elsewhere, but rather lives here.

This is the python function that complies the system image. This is the "button" to push.

And here is the code that loads the system image if present.

The problem with that is that the sysimage is a global choice

I don't think so, or maybe I don't understand. I think you could have Julia installed, as you do, in /.julia/pythoncall/julia-1.6.4/, but have a separate system image created and stored in each python package.

[cjdoris]

By global I meant that when you fire up Julia, it can only use one sysimage. That means that if there are two packages which would like to be able to control the sysimage being used, they will conflict with each other. So there are two options:

1. Have the user explicitly choose which sysimage to use, e.g. via an environment variable.
2. Have the JuliaCall package itself generate and use a sysimage, providing a way for different packages to signal what they need to be compiled into the image.

In the short term, option 1 is really simple to add so I'll do that. So you can provide your users with a script to generate a sysimage and tell them to use the environment variable, if they want to.

Option 2 would be really neat, but will be a fair bit of work. It would certainly make using custom sysimages seamless: packages essentially just provide precompile scripts, and JuliaCall would gather these up and generate a sysimage automatically. So this is on the to-do list, but further down the list.

[cjdoris]

Issues for these two options are #73 and #76 if you'd like to track those.

[jlapeyre]

I think option 1 is a good way to get the ball rolling. It allows python packages that depend on juliacall to experiment with ways to compile and use a system image.

[cjdoris]

Closing, but happy to discuss further.