WIP: generate precompile as part of build process

[KristofferC]

This is a WIP towards fixing #26503. I got inspired on working on it when doing this manually gave such big benefit (#28075).

So what this does is that a part of building the release build, julia is started with a (new) flag that emits precompile statements to STDOUT, a fake REPL (copy pasted from the REPL test code) is spun up, and commands from a file are fed into that REPL. The recorded precompile statements are then stored in a file that the precompile.jl file includes. The sysimg is then rebuilt with those precompile statements.

Things TODO:

• The test REPL uses a FakeTerminal while the real REPL uses a TTY. This means that a lot of the precompile statements that depend on the Terminal are not applicable. Search and replace FakeTerminals.FakeTerminal for REPL.Terminals.TTYTerminal works well enough...
• This now builds the sysimg twice on make. Annoying for development. Not sure how it is best to only do this when you want to make a real release.
• I know almost nothing about Makefile so everything here is just from looking at the surrounding code. Help appreciated.
• This doesn't manage to record everything that the manual version does. We can see what it misses: Update: fixed.

But after running this, startup of julia goes from 0.4s to 0.2s on my computer and first command in the REPL is instant etc. We could also record e.g. the status output for Pkg to make that quick if we want to.

[ararslan]

Not sure how it is best to only do this when you want to make a real release.

We have some logic somewhere (though I can't recall where offhand) that checks whether the current Git SHA matches that of a tagged release. We could use that to conditionally enable this.

[StefanKarpinski]

Wouldn't the tagging happen after the release is built?

[martinholters]

We'd definitely want to do this before tagging a release to verify it's working, no? Maybe do it by default but offer an option to be set in Make.user to disable it?

[KristofferC]

Small update. I fixed the problem that not all things got recorded so this is now just as good as the manual version in #28075.

Seems to have been some improvements on master as well so with this we now have

➜  time ./julia -e 1+1
./julia -e 1+1  0.15s user 0.12s system 105% cpu 0.260 total

➜  time julia-beta -e 1+1
julia-beta -e 1+1  0.41s user 0.16s system 141% cpu 0.400 total

So almost 3x faster startup vs beta. REPL is also very fast (first beta then PR: https://asciinema.org/a/rjvVIEm8gNd8KskeitmJCNPVU).

Improvement is also visible on CI:

 ==> ./julia launch speedtest
0.32user 0.26system 0:00.32elapsed 179%CPU (0avgtext+0avgdata 173540maxresident)k
0inputs+0outputs (0major+22321minor)pagefaults 0swaps
0.32user 0.29system 0:00.30elapsed 199%CPU (0avgtext+0avgdata 173632maxresident)k
0inputs+0outputs (0major+21990minor)pagefaults 0swaps
0.32user 0.28system 0:00.31elapsed 197%CPU (0avgtext+0avgdata 174168maxresident)k
0inputs+0outputs (0major+21971minor)pagefaults 0swaps

vs

 ==> ./julia launch speedtest
0.21user 0.12system 0:00.27elapsed 124%CPU (0avgtext+0avgdata 166500maxresident)k
0inputs+0outputs (0major+22203minor)pagefaults 0swaps
0.21user 0.12system 0:00.26elapsed 126%CPU (0avgtext+0avgdata 168952maxresident)k
0inputs+0outputs (0major+22466minor)pagefaults 0swaps
0.22user 0.21system 0:00.26elapsed 162%CPU (0avgtext+0avgdata 168432maxresident)k
0inputs+0outputs (0major+22163minor)pagefaults 0swaps

---

Maybe do it by default but offer an option to be set in Make.user to disable it?

This is what I have done now. Double building the sysimg on CI is arguably a bit unfortunate though?

Does someone understand why AV can't find the files in contrib:

ERROR: could not open file \cygdrive\c\projects\julia\contrib\generate_precompile.jl

Trying the build bots to see how they fare:
https://build.julialang.org/#/builders/43/builds/1749
https://build.julialang.org/#/builders/83/builds/1786
https://build.julialang.org/#/builders/1/builds/1540

[martinholters]

Double building the sysimg on CI is arguably a bit unfortunate though?

Yes. But we also want this tested.

[KristofferC]

I'll write down the process this uses to get feedback.

• The option of tracing precompile statement was moved from a macro definition to a CLI flag --trace-compile. The macro definition instead sets the default of this flag.
• Another step is added to the target of julia-release and julia-debug which executes last. This runs julia and executes the contrib/generate_precompile.jl script.
• generate_precompile.jl checks if the REPL is available and in that case first starts an interactive seesion with --trace-compile=yes, waits for the repl to load and then exit. It then executes the precompile_replay.jl script (with --trace-compile=yes) which starts a fake repl session and does some common things like access the help, print something, paste something etc. This is right now just defined in a string literal but could also be generated from a REPL session perhaps. If no REPL is available, we just record the startup of a non interactive julia session.
• The new precompile statments are merged into base/precompile_local.jl (which is gitignored).
• A second sysimg build is now kicked of with base/precompile.jl including base/procompile_local.jl.
• The new sysimg now contains all the precompile statments from starting Julia and interacting with the REPL.

[StefanKarpinski]

So it seems like the main question is when to do this during CI builds. On the one hand doing it all the time seems wasteful and somewhat slow, on the other hand not doing it all the time risks this process bitrotting. Is that the consideration or is there something else?

[oscardssmith]

Could this process somehow keep track of the hash of the function, and only re-precompile a function if it changes?

[StefanKarpinski]

That seems roughly equivalent to checking in the generated precompiles and doing an extra sysimg compile round when it changes. I wonder if there’s a way to tell from the diff if the old precompiles will straight up fail or if they will merely be suboptimal?

[vtjnash]

This now builds the sysimg twice on make

FWIW, this is precisely equivalent to building it once, and saving the result after running whatever steps created the precompile.jl file. Or, you can equivalently add a second precompile file into the process during the --output-o sys-o.a phase (after the --output-ji sys.ji phase), and similarly avoid doing any extra or repeated work.

[KristofferC]

Okay I can see how I can probably get the REPL interaction precompile statements without generating the sysimg twice but the ones that come from the actual startup, don't I need an already built julia executable for that?

[staticfloat]

We'd definitely want to do this before tagging a release to verify it's working, no? Maybe do it by default but offer an option to be set in Make.user to disable it?

You would test this by creating a git tag in your local repository, then building Julia and verifying that it all worked. E.g.:

$ git tag v9.9.9
$ echo "9.9.9" > VERSION
$ make

On the one hand doing it all the time seems wasteful and somewhat slow, on the other hand not doing it all the time risks this process bitrotting.

It seems to me that what we really want is almost like a git prehook check that figures out the "optimal" precompile statements dynamically and auto-updates the correct list of precompile statements automatically. If bad precompile statements can break the build, I would argue we should do this generation as a part of every PR at least. If it can't break the build but can only make thing slower, seems like a good candidate for a weekly bot autorun or something.

[KristofferC]

figures out the "optimal" precompile statements dynamically and auto-updates the correct list of precompile statements

This is pretty much what this PR is doing, except it doesn't commit the precompile statements since they are so ephemeral (at least the ones for anonymous functions).

If bad precompile statements can break the build

They can (symbols for anonymous functions disappearing), but we could just try catch them I guess.

I don't see much reason checking these into version control though. Something similar to what we have here where they are an output from the build process itself seems better.

[martinholters]

Okay I can see how I can probably get the REPL interaction precompile statements without generating the sysimg twice but the ones that come from the actual startup, don't I need an already built julia executable for that?

I don't think so. Or rather, no already built sys.so. IIUC, julia --output-o sys-o.a --sysimage sys.ji do_some_stuff.jl should put native code in sys-o.a for anything that was compiled while running do_some_stuff.jl (which may or may not be a bunch of precompile commands), and does not need sys.so.

[KristofferC]

Ok, reworked this so it doesn't require an extra sysimg build and the precompile statements are instead generated when the sys-o.a is being emitted. Overhead generating precompile statements seems to be around 30 seconds compared to the old precompile file (but we compile a bit more now as well). A nice side effect is that the precompile file can be completely deleted so the statements will never go stale.

Comments on the implementation appreciated. The reason why I am launching extra processes in generate_precompile are:

1. It is hard to get a complete trace of the REPL startup procedure that is the same as when we actually launch a julia session without it.
2. I need to do the FakeTerminals -> TTY replacement.

Note that this doesn't actually make startup time much faster (than the old precompile file) because most of the things that took time and was anonymous functions got moved to be lazily loaded.

[KristofferC]

Worked around #28308 by not using mktempdir(::Function). Straight flush again :)

[StefanKarpinski]

Any reason not to merge this right away?

[KristofferC]

Since this touches the build system which feels more fragile for some reason, I think it should be looked over a bit first by those familiar with it.

[KristofferC]

But this has been open for more than a week with few changes and should be very easy to revert since it doesn't touch any files that are really modified so let's try it out on the build bots and if they explode we can just revert. I'm quite excited to try the nightly with this to see if it works well...

[JeffBezanson]

Just tried this. Best thing since sliced bread. The system image isn't even any bigger! Awesome work.

[JeffBezanson]

Now I'm getting greedy:

jeff@gurren:~/src/julia$ ./julia -e 0 --trace-compile=yes
precompile(Tuple{Type{Logging.ConsoleLogger}, Base.TTY})
precompile(Tuple{Type{REPL.Terminals.TTYTerminal}, String, Base.TTY, Base.TTY, Base.TTY})

:(

[KristofferC]

I've seen a few left overs but they seem to be in the ms time range?

They are also not anonymous so could just be explicitly added.

[ararslan]

The output during the build is a bit confusing. What is this telling me?

Sysimage built. Summary:
Total ─────── 163.782677 seconds
Base: ───────  42.277126 seconds 25.8129%
Stdlibs: ──── 121.503334 seconds 74.1857%
    JULIA usr/lib/julia/sys-o.a
Generating precompile statements...
──────────────────────────────────────
Hello World
  0.000003 seconds (32 allocations: 2.313 KiB)
search: reinterpret

/Users/alex/Projects/julia/base
julia> ──────────────────────────────────────
736 precompile statements generated in  45.842150 seconds
    LINK usr/lib/julia/sys.dylib

[KristofferC]

You mean the stuff in between the bars. It's just some output to stdout from the REPL replayer. Could probably be made more quiet. But it needs TTY for stdout which is why I am not redirecting it to a file or Pipe.

[JeffBezanson]

Hm, we also still seem to get the uv_close assertion failure sometimes (just got it on freebsd CI). Looks like an invalid libuv handle type? Maybe something is trying to close an uninitialized object?

[JeffBezanson]

Why does generate_precompile.jl explicitly call Base.__init__()? Seems like a strange thing to do.

[KristofferC]

Because it needs libuv events to be setup to be able to use things like pipes.