Move compiled cache from DATAROOTDIR to LIBDIR

[nalimilan]

The FHS specifies that only architecture-independent files should be put under /usr/share (DATAROOTDIR). Originally we only stored .ji files in the "compiled" directory, but now we also put .so files.

Keep /usr/share and /usr/local/share in the default DEPOT_PATH for backward compatibility and because they can still be used for architecture-independent files such as environments or registries.

Also fix a few uses of "data" which should have been DATAROOTDIR and wouldn't work if using a different DATAROOTDIR when building.

[nalimilan]

I'm actually not sure it's a good idea to look in /usr/local by default. I've kept /usr/local/share/ for backward compatibility, and added /usr/local/lib for consistency. But this doesn't make a lot of sense as the next two dirs (4 and 5) are relative to the julia path so they will correspond to /usr/local/ if Julia was installed there. And if there are two Julia copies, one in /usr and the other in /usr/local, the latter is going to interfere with the former.

[tecosaur]

While I wish otherwise, my current impression is that Julia doesn't care much about Linux norms. If anything, setting the depot to /opt/julia may be the most appropriate suggestion.

The UNIX business of spreading all the files related to a single application across dozens of directories that no one can keep straight is absolute and utter nonsense. — Stefan, JuliaLang/juliaup#844 (comment)

[nalimilan]

While I wish otherwise, my current impression is that Julia doesn't care much about Linux norms.

That's not really the case. I've made lots of PRs to ensure Julia can be built and installed to follow the FHS, and so far it was respected. This is a bit different from XDG_CONFIG_HOME, as it doesn't mean the directory will change its location from one system to the next (and users generally don't touch the built-in depot).

If anything, setting the depot to /opt/julia may be the most appropriate suggestion.

/opt is generally organized with one directory per app, precisely to avoid spreading files. Users can already do that (and probably do) by extracting the binary tarball to /opt/julia.

[tecosaur]

That's not really the case. I've made lots of PRs to ensure Julia can be built and installed to follow the FHS, and so far it was respected.

I'm glad to hear, my interaction has been mainly around XDG and the concerns I've seen put forward in my interactions (scattering is awful, we want to do the same thing on every OS and that's more important than the OS's conventions, working out what goes where is too complicated, etc.) are comments that apply pretty equivalently to FHS and XDG's Base Directories, particularly given the number of direct equivalents:

• /etc (system) = XDG_CONFIG_HOME (user)
• /var/cache (system) = XDG_CACHE_HOME (user)
• /usr/share (system) = XDG_DATA_HOME (user)
• /var/run (system) = XDG_RUNTIME_DIR (user)

And when I see Stefan say "The UNIX business of..." I presume he's referring to the FHS.

/opt is generally organized with one directory per app, precisely to avoid spreading files

Right, which is precisely why I suggested it given the "put everything in one place" / "spreading files is awful" comments I've seen in response to suggestions that Julia should respect the XDG Base Directory spec.

Anyway, this PR seems sensible and IMO the more considerate Julia can be of the host system's norms the better 🙂.

[nalimilan]

I think the main difference with XDG is that following the FHS doesn't spread files by default, as you can extract the binary tarball and keep contents in a single place (as opposed to calling make install). Stefan is against following XDG by default but is OK with having it opt-in. This is what we do for the FHS (and in this PR).

[nalimilan]

The FreeBSD CI failure seems related but I can't figure out why it happens only on that platform. Before I investigate this further I'd like to make sure people are OK with the current approach of this PR. With these changes the compiled dir is under usr/lib/julia while the sources are still under usr/share/julia, and these paths are two different entries in DEPOT_PATH. In practice this doesn't seem to be a problem (and it's in the spirit of the FHS). Another solution would be to move both sources and compiled images to usr/lib.

Opinions? @KristofferC maybe?

[KristofferC]

Seems a bit unfortunate to have to "bloat" the depot path so I guess only based on that moving the sources as well would be preferable? I know very little about these linux paths so can't comment on that part.

[nalimilan]

The FHS isn't super explicit:

/usr/lib includes object files and libraries (Footnote: Miscellaneous architecture-independent application-specific static files and subdirectories must be placed in /usr/share.)

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s06.html

The /usr/share hierarchy is for all read-only architecture independent data files.

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html#idm236091847456

Actually I've realized Python had a discussion about the same issue a long time ago (python/cpython#36954) and they concluded source files can be considered as libraries. And indeed nowadays in Fedora and Debian Python packages install both .py and .pyc under /usr/lib. And so does R. So let's do that too, I'll update the PR.