IJulia broken on update — re-build IJulia when updating Julia?

[stevengj]

When julia is updated, juliaup changes the installed directory location of the julia binary. This breaks IJulia until it is re-built, because IJulia installs a kernel configuration file that refers to a specific Julia path. For example, when updating from 1.10.0 to 1.10.1 on my Mac, trying to open a Julia notebook in Jupyter led to an error:

[Errno 2] No such file or directory: '/Users/stevenj/.julia/juliaup/julia-1.10.0+0.x64.apple.darwin14/bin/julia': '/Users/stevenj/.julia/juliaup/julia-1.10.0+0.x64.apple.darwin14/bin/julia'

Should juliaup update automatically re-run Pkg.build("IJulia") if it changes the location of the Julia binary, assuming IJulia is installed? (This will also update the kernel identifier in the Jupyter menu, e.g. from "Julia 1.10" to "Julia 1.10.1".)

Or should IJulia be installing a Julia kernel with a different path for julia somehow when juliaup is being used? Currently, it uses joinpath(Sys.BINDIR, "julia")

[StefanKarpinski]

Obviously, this shouldn't be IJulia-specific, there should be a mechanism for fixing any package that has the same issue. Is there something special about what IJulia does that causes this to break?

[KristofferC]

It puts an absolute path to the Julia executable at a place. It could probably have put julia +1.9 in there instead. The app stuff in Pkg I am working on had an identical design issue with storing the julia executable used to run the app.

[davidanthoff]

At least on Windows that has always been a problem, I think? I think the old Julia Windows installer installed into a path that had the patch version in the folder name, so one had to rebuild IJulia every time one installed even just a patch update. On Mac that was different because (I think) the old installer installed into a location that only contained the minor version in the folder name. But then of course one couldn't install multiple patch versions of Julia in parallel, I guess?

But regardless, it would be good to make this smoother for end-users... In VS Code the whole kernel detection is solved much nicer: one essentially can register something like "Julia will provide kernels", and then VS Code will query the Julia extension at every startup which kernels it wants to provide. So there is no configuration file that needs updating or anything like that. Would be interesting to see whether Jupyter provides a similar "dynamic kernel detection" mode these days? That would be the most elegant solution, I think.

I think @KristofferC's suggestion is probably the next best idea, i.e. don't store a full path to the Julia executable but something like julia +channelname. We could for example add an env variable JULIAUP_CURRENT_CHANNEL that is always set if Julia was started via Juliaup and holds the channel name, then IJulia.build could check for the existence of that env variable, and then know whether it is running in a Juliaup context and what kind of command line it should write.

I think there is also a bit of a deeper issue here: once we implement Julia version selection via the manifest in Juliaup, we'll have this weird situation that both a notebook and the manifest might store a Julia version, and then it becomes unclear which is more important, and how that then interacts with registered kernels... But maybe a problem for later to solve.

[stevengj]

Is there a way to detect (from within Julia) that juliaup is being used, and the name of the current channel? This would be necessary if we wanted to use julia +channelname in the IJulia build script when creating the Jupyter kernelspec file.

[davidanthoff]

Not at the moment, my paragraph 3 above is essentially a suggestion to achieve that.

[stevengj]

The problem with environment variables is they are inherited by subprocesses. If I do run(`/path/to/some/other/julia`) from within the juliaup-launched process then the new julia subprocess will still have JULIAUP_CURRENT_CHANNEL set, even though that is no longer correct.

It would maybe be better to set a global variable in Julia itself. Couldn't it define something in the Sys module, for example, e.g. defining Sys.JULIAUP_CHANNEL for the channel and Sys.JULIAUP_BINDIR for the .juliaup/bin directory that julia was invoked from (so that we can get an absolute path of .juliaup/bin/julia, rather than assuming that it is always in the global PATH)?

[davidanthoff]

The problem with environment variables is they are inherited by subprocesses. If I do run(/path/to/some/other/julia) from within the juliaup-launched process then the new julia subprocess will still have JULIAUP_CURRENT_CHANNEL set, even though that is no longer correct.

Yep, true. I think (maybe) the philosophy of Juliaup is that one should always start any Julia process via the Juliaup launcher, in which case that problem would be solved because it would always reset that env variable. But not sure that is the right way to think about it...

It would maybe be better to set a global variable in Julia itself.

Hm, maybe? How would that be set, though? And also, that would only work for new Julia versions, it would be nice if we found a solution that also worked on old versions...

Maybe the launcher could set JULIAUP_CURRENT_CHANNEL, and then newer Julia versions could detect that at startup, set a global variable and unset the env variable? But then we would have a different behavior on different Julia versions, which also isn't great...

[davidanthoff]

Maybe long-term we should go with something completely crazy/different for IJulia: do away with version specific kernels entirely, just have one Julia kernel registered with IJulia. Instead of getting installed as a package, it could be an app in the sense that @KristofferC is working on. And then this one version-agnostic kernel would multiplex the right Julia version at runtime via Juliaup.

[stevengj]

Hm, maybe? How would that be set, though? And also, that would only work for new Julia versions, it would be nice if we found a solution that also worked on old versions...

You can set it in the launcher via Sys.eval(:(foo = bar)).

And then this one version-agnostic kernel would multiplex the right Julia version at runtime via Juliaup.

How is that different from a kernel that runs julia with no arguments?

[davidanthoff]

Hm, maybe? How would that be set, though? And also, that would only work for new Julia versions, it would be nice if we found a solution that also worked on old versions...

You can set it in the launcher via Sys.eval(:(foo = bar))

The launcher is a small binary program written in Rust. On Windows that then launches a specific Julia version as a sub process, on Mac/Linux it forks (if that is the right terminology...) and somehow replaces itself with the right Julia process. But in neither case can it run Julia code...

And then this one version-agnostic kernel would multiplex the right Julia version at runtime via Juliaup.

How is that different from a kernel that runs julia with no arguments?

The version agnostic kernel could look at either the notebook file and what Julia version was stored there and then launch the appropriate Julia version, or the manifest of the project where the notebook is located and read the Julia version from there.

[stevengj]

The version agnostic kernel could look at either the notebook file

The kernel doesn't have access to a notebook file. A notebook file may not even exist. The kernel just talks to a server that is sending it snippets of code to execute (etc.) and sends back the results.

When you are using Jupyter notebooks, the notebook file says what kernelspec to launch, so this is the place where the version (if any) must be encoded.

[davidanthoff]

Argh, you are of course right! I'm just thinking mostly in the VS Code notebook API these days, where all of this is so much easier...

So I think that means we just are back to the idea that the IJulia build script needs a way to know what Juliaup channel it should use... I'm still stuck with the env variable as the only idea, but happy to consider other options as well.

[stevengj]

Why can't the launcher spawn the Julia process with a `-e 'Sys.eval(:(JULIAUP_CHANNEL = "foo"))' argument or similar, rather than using an environment variable (which is problematic because it is inherited as I mentioned above)?

But environment variables would be better than nothing. This is biting people on every release.

[davidanthoff]

Why can't the launcher spawn the Julia process with a `-e 'Sys.eval(:(JULIAUP_CHANNEL = "foo"))' argument or similar

We would have to merge that with a user-provided -e argument, I think? If we can do that reliably, then that seems like a good idea to me.

[stevengj]

No, you can pass multiple -e arguments:

% julia -e 'println("Hello")' -e 'println("world.")'
Hello
world.

However, it looks like a -e argument can't be combined with a script argument? julia -e 'println("Hello")' foo.jl does not seem to execute foo.jl. 😢 I guess you could convert it to julia -e 'println("Hello")' -L "foo.jl", but I think that doesn't pass arguments correctly. 😢

Update: it looks like you can do:

julia -L /path/to/juliaup_sys.jl …

where juliaup_sys.jl is a file that contains your eval statements to set Sys.JULIAUP_CHANNEL or whatever, and -L arguments (unlike -e) will correctly compose with other arguments, including script files (or other -L args).