How to use with PackageCompiler?

[nlw0]

It would be great to add JuliaSnail.jl to a sysimage created with PackageCompiler. I'm not sure how to do it, though, as the module is just included, as far as I know, and PackageCompiler.create_sysimage expects a list with module names. Does anybody have any tips?

[gcv]

I failed to make my own custom images before, though TBH I timeboxed the task to 15 minutes and didn't try that hard. If someone can post an ELI5 level set of instructions, I will be glad to include them in the Snail documentation.

[nlw0]

To be able to do it we need to find a way for packagecompiler to load the module, but it might require also running a few functions. I have no idea, though. Is there a way we can run a couple of functions simulating what julia-snail does during a session?

[nlw0]

Alternatively, it might be the case that most of the start-up time for snail is actually some dependency such as CSTParser, is this possible?

[gcv]

It's definitely dependencies, but the JIT compiler probably needs to have seen Snail functions to optimize them. A great start is to look through julia-snail.el and find calls made to the JuliaSnail module.

[nlw0]

It might help doing this if JuliaSnail.jl was more like a module that we install normally. I understand that now it's loaded by includeing the source and importing Main.JuliaSnail. Is there any chance we could change that? I bet it would make it easier to work with PackageCompiler, although I'm not sure it's strictly necessary. I tried to install JuliaSnail but the Packages.toml is missing a name attribute.

Anyways, if dependencies such as CSTParser are mostly responsible for the start-up time, then just having that compiled might be enough and easier to accomplish. I would be very happy to reduce the start-up time, I tend to restart the shell often because I never learned to work otherwise.

[nlw0]

Now I see JuliaSnail.jl contains multiple sub-modules. If it's not easy to make the whole of JuliaSnail into an installable module, we might try just taking this CST sub-module, make it a separate installable "JuliaSnailCST.jl", and we package-compile that. You even have this forcecompile function that is probably exactly what you need for precompile_execution_file in a call to PackageCompiler.create_sysimage.

[gcv]

Making any part of Snail installable through normal Julia packaging channels introduces headaches. Version compatibility checks, extra steps to cut releases, extra installation steps for users, undoubtedly more support questions, and generally more work for me. So I'm not very interested in doing that.

Is making an installable module really a prerequisite for PackageCompiler?

[nlw0]

I certainly don't mean to turn it hard to install. First step is just figuring out a way to do it, that's just one idea... I don't know enough about the PackageCompiler or JuliaSnail inner workings to be sure of what's needed. I'm not sure installing the package is a required thing to do, but my hunch is it would make it much easier.

If compiling this CST sub-module is really the heart of the matter, I think it's a good idea to try that out. Then if it works we can look for a way to make it easier. I'll try to investigate that and get back to you.

[nlw0]

@gcv I did it! I got JuliaSnail.jl installed in my default Julia environment and used PackageCompile to create a sysimage along with Makie. Now I can start a new Julia session and see my first plot very quickly. What a bliss!

I would love to help others do the same thing. I believe it is possible to have the automatic and simple initialization of JuliaSnail.jl, similar to using -L, but relying more on the package management system. The first way I can think is through the environment variable JULIA_LOAD_PATH. Not sure there may be a limitation, but looks promising to me. I feel it might be very helpful to the project to investigate that more.

I created a fork that contains some modifications I had to do. First of all, I needed to add a couple of dependencies, and the package wasn't installable as well because there was no name or uuid. I've also removed the 'with_dir_env` function just to see if we can make things work without it, but I haven't tested the plugins yet. https://github.com/nlw0/julia-snail/tree/nic/pkg-structure

[gcv]

Can you explain what you did in detail?

[nlw0]

Here's a tentative guide:

Make the project compatible with Pkg

The project structure right now is not recognized by Pkg:

Some modifications are necessary for that to work. Either checkout the branch https://github.com/nlw0/julia-snail/tree/nic/installable-package, or do the following:

1. Move JuliaSnail.jl to src/
2. Project.toml needs a name and uuid fields. To obtain a uuid you can create a new empty project with Pkg.generate and steal the value from the new Project.toml. The authors field is optional.
3. Include missing dependencies (Base64, Markdown, Printf, REPL, Sockets, apart from CSTParser). This can be done via julia --project=julia-snail and Pkg.add.
4. On src/JuliaSnail.jl it is necessary to either comment out the import Pkg, or move it within the module definition, in which case it also needs to be listed as a dependency.

After these modifications it should now be possible to either:

1. Install the package via Pkg.add or eg. ] dev "~/src/julia-snail".
2. Run JULIA_LOAD_PATH=:~/snail/julia-snail julia and call using JuliaSnail within Julia. In this case the package is not installed in the primary environment, but is found through LOAD_PATH.

At this point, if you install it, you can remove the "-L" from the Julia call at julia-snail.el. Not saying this is how it should work in the future, this is just for experimental and illustrative purposes. julia-snail should now be able to open a Julia terminal and load the installed JuliaSnail module.

Create a sysimage

Once the package can be found via Pkg, it can be easily added to a sysimage by following the usual PackageCompile instructions. You can add more packages as well. Here's one way to do it.

Create the script preco.jl

using OhMyREPL
import JuliaSnail
import GLMakie
import ImageView
import Images
JuliaSnail.start()
JuliaSnail.CST.forcecompile()
img = Images.load("./buska.png")
ImageView.imshow(img)
f=GLMakie.Figure()
GLMakie.image(f[1,1],img)
GLMakie.lines!(f[1,1],randn(11))
GLMakie.scatter!(f[1,1],randn(11),randn(11))
GLMakie.save("test.png",f)

Run julia --startup-file=no --trace-compile=myprecostatements.jl and run the script commands. Running Julia interactively may be necessary to trigger OhMyREPL. Otherwise just running julia preco.jl --startup-file=no --trace-compile=myprecostatements.jl non-interactively should work as well.

Once the precompile statements file is created, you can call julia again and run

using PackageCompiler
PackageCompiler.create_sysimage(["JuliaSnail", "GLMakie", "ImageView", "Images", "OhMyREPL"];
    sysimage_path="MySysimage.so", precompile_statements_file="myprecostatements.jl")

Notice this is different from the precompile_execution_file approach, which might work as well, I'm not sure.

After a few minutes your sysimage file should be created, and calling Julia with -J MySysimage.so should give you a very acceptable time-to-first-plot!

[gcv]

That's a great guide. Thanks for writing it up.

I'm now thinking about adding a "wizard" to Snail that would automate a bunch of steps for users building custom images. You added GLMakie and ImageView, for example, and other users might want to add other things.

It sounds like we can't make an image based purely on the current state of the Julia REPL, and PackageCompiler has a hard requirement for anything bundled into an image to be a Julia "package"? It's surprising and annoying, but I guess we'll live with that.

Do you know why all those explicit dependencies have to be added (Printf, REPL, Sockets, etc.)? They're built into Julia itself.

[nlw0]

Happy to help! i've been using julia-snail for a while now, looking for ways to contribute.

I think the only module you can really consider built-in in Julia is Base, everything else are just like other modules. Pkg is a little fussy with the dependencies, etc, but this is supposed to help! Same for PackageCompiler, I think when you start trying to do things around the way it works there will be some nasty surprises.

It would be nice to have some easier way to create a sysimage, but on the other hand people tend to want lots of customization, like setting up emacs! Maybe as we come up with a good procedure and documentation first, we will naturally reach a point we can offer a simple script for the basics.

[gcv]

I can't get past this error: ERROR: package(s) JuliaSnail not in project. It happens when I run this:

PackageCompiler.create_sysimage(["JuliaSnail"], sysimage_path="img1.so", precompile_statements_file="precompile_statements.jl")

Note that import JuliaSnail succeeds in the same REPL:

$ JULIA_LOAD_PATH=:~/Code/julia-snail julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.2 (2022-02-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> import JuliaSnail

julia> import PackageCompiler

julia> PackageCompiler.create_sysimage(["JuliaSnail"], sysimage_path="img1.so", precompile_statements_file="precompile_statements.jl")
ERROR: package(s) JuliaSnail not in project

There's no precompiling output for JuliaSnail and PackageCompiler because I had run this before.

I made a src directory and symlinked JuliaSnail.jl inside it. My Project.toml has a uuid entry and I added the extra deps entries. With JULIA_LOAD_PATH set as you noted, using JuliaSnail works. precompile_statements.jl contains a boatload of precompile calls generated as you noted from a precompile.jl file with --trace-compile.

[gcv]

(Updating this to keep track of comments on the Julia Discourse.)

From https://discourse.julialang.org/t/ann-julia-snail-an-emacs-development-environment-version-1-1-0-release/65980/9:

the command julia preco.jl --startup-file=no --trace-compile=myprecostatements.jl does not produce any precompile output, at least for Julia 1.7.3, because the command-line switches must appear before the script name, otherwise they get interpreted as command-line arguments for the script instead of Julia itself. So you need instead julia --startup-file=no --trace-compile=myprecostatements.jl preco.jl

I'll have to give this another go.

[nlw0]

Hi @gcv , apologies I didn't come back to this earlier. I'm upgrading to Julia 1.8.3 now. I was dreading a bit trying to do the process again, but I'm happy to say that following my (already forgotten) guide I was able to pull it off. I'd love to help more people be able to do it, and continue to make the process easier, because it's still such a big deal to make sure we have a good time-to-first-plot, especially for users of "alternative" IDEs. Theoretically, the tools should be already there.

It seems there might still be some detail involving either the installation/availability of JuliaSnail, or some relationship between package availability and PackageCompiler. I notice you just imported the package in that example. Does it actually run if you using it? Or for instance, is this running flawlessly?

import JuliaSnail
JuliaSnail.start()
JuliaSnail.CST.forcecompile()

Anyways, I would suggest trying out actually installing it with Pkg.add, which is what I usually do, and see if we can make at least that work, then we can figure out where the difference might be.

[gcv]

@nlw0: Cool, thanks for following up. I’ll try to find time in the next few weeks to pick this up again. I agree that TTFP is pretty essential. I think I already mentioned that I’d love for Snail to provide “make an image out of my current project” feature.

[gcv]

@nlw0: Have you had a chance to try the latest Julia 1.9.0 prereleases? They seem to have made major improvements on the time-to-first-plot problem. If this problem will simply go away in the short to medium term, I'd rather not invest time into PackageCompiler integration.

[chriselrod]

The advantage to PackageCompiler is that it solves using time, while the 1.9 improvements do not.