use Preferences

[t-bltg]

@jheinen, I though #465 was reworked to implement the usage of Preferences, but it was obviously not.

So this is a proposal to fix JuliaPlots/Plots.jl#4334 (comment), allowing to no use GR_jll artifacts at all, and use a system installation instead, using optional Preferences.jl configurations.

2924a1d is not acceptable since explicitly discouraged by the following error message:

ERROR: LoadError: InitError: Evaluation into the closed module `GR` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `GR` with `eval` during precompilation - don't do this.

It is still unclear what are you scenario usage, so please do comment on what is needed / missing from the following cases.

case 1: default

julia> using GR  # use the binaries pointed to by `GRPreferences`, defaulting to `GR_jll` if `LocalPreferences.toml` does not have `GRPreferences` entries

case 2: force usage of GR_jll artifacts

julia> using GRPreferences

julia> GRPreferences.use_jll_binary(; force=true)

julia> read(joinpath(Base.active_project() |> dirname, "LocalPreferences.toml"), String) |> println
[GRPreferences]
__clear__ = ["grdir"]
binary = "GR_jll"

[MPIPreferences]
[...]

julia> using GR  # uses artifacts from `GR_jll`

case 3: force usage of local gr installation somewhere on the system

julia> GRPreferences.use_system_binary("/opt/gr"; force=true)  # I've built and installed a local gr in "/opt/gr"

julia> read(joinpath(Base.active_project() |> dirname, "LocalPreferences.toml"), String) |> println
[GRPreferences]
binary = "system"
grdir = "/opt/gr"

[MPIPreferences]
[...]

julia> using GR  # uses system installation, does not use `GR_jll`

case 4: no write access to LocalPreferences.toml, GRDIR via env var

$ GRDIR="/opt/gr" julia
julia> using GR

julia> @show GR.GRPreferences.grdir[] GR.GRPreferences.libGR[]
GR.GRPreferences.grdir[] = "/opt/gr"
GR.GRPreferences.libGR[] = "/opt/gr/lib/libGR"

julia> using GR  # uses system installation, does not use `GR_jll`

Side note: why does this repo not have a proper CI workflow for PRs ?

Note that we don't need to add libraries extensions (.so, .dylib, .dll) since dlopen will handle that automatically.

[t-bltg]

Local tests with GR_jll or system binaries and libraries are ✔ on linux.

Please approve the workflows for ci to run.

[jheinen]

Many thanks for this extensive PR.

Unfortunately, I don't see how to integrate this change into our environment. The point is that we use a common Julia distribution for all users (~1000), which in turn is distributed via a (read-only) network drive for different operating systems. Users no longer have to worry about software or package updates - we manage this centrally on a daily basis. We proceed similarly with other systems, e.g. Python. We have been using this approach for years and it has worked very well. Obviously, this contradicts the Julia logic regarding the package management.

My tests have shown that using #471 requires write access to the package repository. In addition, to change the run-time, you must first load or modify the corresponding GRPreferences. At least I haven't found out how to control this before Julia starts (e.g. via an environment variable). But maybe I did something wrong.

I don't want to contradict the snoop compile logic in any way. I will reverse 2924a1d in any case. If all else fails, I will have to work with two Julia depots. If you have a better idea, please let me know.

[t-bltg]

We have been using this approach for years and it has worked very well. Obviously, this contradicts the Julia logic regarding the package management.

I see, having a centralized depot is not really uncommon. I myself use that across multiple julia versions to avoid using too much space.

My tests have shown that using #471 requires write access to the package repository

I didn't take permissions into account. Let me think of something.

At least I haven't found out how to control this before Julia starts

I think that this can still be achieved via an env variable ENV["GRDIR"] in GRPreferences: I'll reintroduce that.

If all else fails, I will have to work with two Julia depots. If you have a better idea, please let me know.

Worst case scenario. I'm sure there is another way to tackle this problem. I'll rework that now.

[jheinen]

I'm sorry for taking up so much of your time. But I didn't get deep enough into the snoop compile mechanism yet.

[t-bltg]

From my tests this works now, can you confirm ?
import GR_jll is now in a try / catch block to prevent loading failure when ENV["GRDIR"] is set (per JuliaPlots/Plots.jl#4334 (comment)).

EDIT: Plots, and GR tests are ✔ with or without GR_jll.

[jheinen]

On macOS (with GRDIR set to /usr/local/gr) GR tries to start GKS QtTerm instead of GKSTerm:

env: /usr/local/gr/bin/gksqt: No such file or directory

I did not yet test on Linux, but I'm afraid there will be problems since GR_jll is loaded in both cases.

[t-bltg]

You mean this GKSTerm ?

GR.jl/build_tarballs.jl

Line 60 in 9ed2c4f

ln -s ../Applications/GKSTerm.app/Contents/MacOS/GKSTerm ./

What controls usage of GKSTerm over gksqt on macOS https://github.com/JuliaBinaryWrappers/GR_jll.jl/blob/bea8372f24216f3729c99c814b7c063054f9c0ec/src/wrappers/x86_64-apple-darwin.jl#L49 ?

I did not yet test on Linux, but I'm afraid there will be problems since GR_jll is loaded in both cases.

I don't think so, since even if GR_jll manages to load in GRPreferences (allowed to fail), we dlopen a different shared library in

GR.jl/src/funcptrs.jl

Line 29 in 3693598

libGR_handle[] = Libdl.dlopen(GRPreferences.libGR[])

[jheinen]

The decision about GKSTerm is made at compile time in utils.h.

I think, we also have to manage libGKS.* in preferences.jl.

[t-bltg]

The decision about GKSTerm is made at compile time in utils.h.

Yes, but where it is made in GR.jl ?

I think, we also have to manage libGKS.* in preferences.jl.

I see, done.

[t-bltg]

So we still need to settle the GKSTerm thing on macOS.