WIP: App support in Pkg

[KristofferC]

This is quite heavily WIP towards having "app" support in Pkg. An app is a program that you just write its name in the terminal and it starts up, without explicitly having to invoke Julia, load the package, and call a function. Every app has an isolated environment.

More details of the design can be found in this hackmd: https://hackmd.io/r0sgJar5SpGNomVB8wRP_Q

This PR requires JuliaLang/julia#52103

Here is some example usage:

(Pkg) pkg> app st

shell> rot13
zsh:1: command not found: rot13

(Pkg) pkg> app add https://github.com/KristofferC/Rot13.jl
    Updating git-repo `https://github.com/KristofferC/Rot13.jl`
  Activating project at `~/.julia/environments/apps/Rot13`
  No Changes to `~/.julia/environments/apps/Rot13/Project.toml`
  No Changes to `~/.julia/environments/apps/Rot13/Manifest.toml`

(@Rot13) pkg> app st
[43ef800a] Rot13 v0.1.0 `https://github.com/KristofferC/Rot13.jl#master`:  rot13 /home/kc/julia/usr/bin/julia 

shell> rot13 Rotate this please
Ebgngr
guvf
cyrnfr

(@Rot13) pkg> app rm rot13
[ Info: Deleted app rot13

(@Rot13) pkg> app st

cc @MasonProtter, @Roger-luo

[tecosaur]

Oh this looks very cool! Thanks for all the time/effort that's gone into this 🤩

---

One thing I'm slightly concerned about here is the approach taken to making sure that the executables are on the users's PATH on Linux/BSD systems.

• There are many more shells than are hardcoded in get_shell_config_file, for instance I know multiple people using elvish, nushell (there was mention of this one on the Julia zulip too) as well as xonsh and oil to name a few. Also, isn't the fact we're relying on a hardcoded list a sign that there's something dodgy about this approach?
• I've seen some shell configurations that have early-exit conditions, which mean that programmatically appending to the shell rc may not actually do anything
• The hardcoded shell file paths aren't actually correct for the supported shells. For example, I can tell you that despite using zsh the current implementation would fail to work on my machine, since I have no ~/.zshrc. If the code changed to create a ~/.zshrc it would be ignored, since I have set my ZDOTDIR to ${XDG_CONFIG_HOME:-$HOME/.config}/zsh.
• Setting the PATH in shell rc files doesn't affect non interactive shell sessions. If we want Julia apps to extend to say graphical apps, this is particularly relevant as usually the launcher process is a child of the desktop environment, and either does not load the shell rc file or only loads it once at login. For this, you also want to potentially modify the shell env/profile/login shells. However, changes to those configurations are only loaded at login, so you'd also need to get the user to log out and log back in again for it to take effect.
• Somebody on the Julia Slack recently mentioned needing to give juliaup write permissions to their shell config in order to successfully install juliaup, and with one particular system configuration needing to broaden write permissions to all users.

I see in the design document there is some mention of putting such files in a more standard location already on the path such as ~/.local/bin, and that Cargo is mentioned in response to this. I think it's worth noting that there is a well-documented series of efforts (like this issue) to make Cargo more XDG-compliant (https://poignardazur.github.io/2023/05/23/platform-compliance-in-cargo/ does a good job outlining this, and describing a path forwards for Cargo). The Cargo discussion can essentially be summed up as "would have been good, but a bit late now".

Other lang's package managers already install things in the XDG-appropriate locations, such as Python with pip install --user (new/alternative Python package managers like poetry copy this behaviour).

I'd advocate for a ~/.local/bin approach on Linux/BSD for these reasons. To programmatically determine which executables in ~/.local/bin are managed by Julia, the executable files could be put inside a Julia-managed directory, and then symlinked to ~/.local/bin. I think this approach keeps much of the benefits of the custom-bindir added to PATH approach while avoiding the major pitfalls.

(NB: when I say ~/.local/bin I really mean ${JULIA_BIN_DIR:-${XDG_BIN_DIR:-$HOME/.local/bin}}, but that's a bit of a mouthful)

[tecosaur]

On a briefer note, shouldn't the unix shim be based on sh not bash, possibly even #!/bin/sh over #!/usr/bin/env sh (somebody else should check, but IIRC that's the POSIX form)?

[tecosaur]

"Somebody else" is apparently me 😄, and my "IIRC" was wrong.

I've just had a look at https://pubs.opengroup.org/onlinepubs/009695399/utilities/sh.html and under APPLICATION USAGE there's this relevant excerpt:

Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH , ensuring that the returned pathname is an absolute pathname and not a shell built-in.

[tecosaur]

From reading around a bit more, it seems like the consensus on portability goes something like this (from most portable to least):

• A binary executable that doesn't have a shebang
• #!/bin/sh
• #!/usr/bin/env sh
• #!/usr/bin/env bash
• #!/bin/bash

[Roger-luo]

nice work! I'm wondering how apps are shared across Julia versions? e.g. are they isolated by Julia versions like how the global environment are setup?

[PallHaraldsson]

intended to be run by the user as appname args to app. [..] It’s assumed that Julia is installed and serves as the “driver” to start up the app.

This seems useful, maybe already dispute that limitation. Could it be lifted by autoinstalling Julia (runtime, right version) for you if not available? Needs not be in first version.

This is in some ways similar to Python's zipapps (which I believe is not too popular, because runtime can't be assumed, even for Linux where it's most often preinstalled), that needs separate .pyz[w] file ending, and Python installed (and are in one archive file, optionally compressed):

https://docs.python.org/3/library/zipapp.html

There is no way to say “python X.Y or later”, so be careful of using an exact version like “/usr/bin/env python3.4” as you will need to change your shebang line for users of Python 3.5, for example.

[We already have AppBundler.jl if you want to bundle the runtime, it's best if you can have one way to make an app and it can be compiled with PackageCompiler, or use AppBundler, or a combining those..., or this system. ]

[KristofferC]

One thing I'm slightly concerned about here is the approach taken to making sure that the executables are on the users's PATH on Linux/BSD systems.

With regards to XDG there is an argument that Pkg should follow what Julia itself does. (As you are aware) there is JuliaLang/julia#4630. juliaup also uses this method of installing Julia and since juliaup is more or less the official way to install Julia it feels like if you have managed to install Julia itself, this should be fine. So there is a tension here between doing XDG (which some people would argue is the correct way) and to fit in how things are done everywhere else in Julia and its ecosystem.

A related question, according to XDG where should the .julia/environments/apps/Package folder go?

For Windows the Cargo issue comment says:

For Windows, everything should go in ~/appdata/locallow or ~/appdata/local,since ~/.cargo is just a cache, AFAICT. This is FOLDERID_LocalAppData for SHGetKnownFolderPath, CSIDL_LOCAL_APPDATA for SHGetFolderPath, and %LOCALAPPDATA% in the environment.

How is that translated to all the files used here (shims, AppManifest.toml, app environments)?

---

Other lang's package managers already install things in the XDG-appropriate locations, such as Python with pip install --user

I get

❯ pip install --user httpie                       
Requirement already satisfied: httpie in /Users/kristoffercarlsson/Library/Python/3.9/lib/python/site-packages (3.2.2)

~/Library/Python/3.9/bin
❯ ls
git-filter-repo  http  httpie  https  markdown-it  pygmentize

[KristofferC]

nice work! I'm wondering how apps are shared across Julia versions? e.g. are they isolated by Julia versions like how the global environment are setup?

As it is right now each app entry in AppManifest.toml has an absolute path to a Julia installation. If you want to update that Julia version you would also resolve the environment. This ties into this later comment:

Could it be lifted by autoinstalling Julia (runtime, right version) for you if not available? Needs not be in first version.

one plan forward is to use Juliaup to install the Julia installation that the app is currently configured for if it does not exist. That way you would not store the absolute path to the julia installation like that.

[tecosaur]

With regards to XDG there is an argument that Pkg should follow what Julia itself does.

Right. I basically see Julia as currently being in a similar situation to Cargo — in that by the end of JuliaLang/julia#4630 I think I can fairly summarise the consensus as "yes this would be nice to have, but it's going to be a hassle to start using it".

Much of the value of the XDG Desktop spec comes via a network effect. Thus when the Desktop spec was new and that issue was created in 2013, the benefit was somewhat speculative. Now though, as more tools use and assume XDG compliance, it creates a growing tension between the "Julia way" and the XDG way.

In this sort of light, I see decisions like this as opportunities to choose between digging down and digging out 😛 somewhat. I still have loose plans to go back to JuliaLang/julia#4630 to see if I can help move the state of affairs closer to XDG compliance (Stefan asked me if I'd be interested in putting a PR together a few months ago, and I am once I have fewer PRs currently open).

Considering the current "Julia way" and the XDG spec, would it not be possible to put things in ~/.julia/bin as the "Julia-managed directory" that executables are written into, and make symlinks into ~/.local/bin? I might well be missing something, but it seems to me that this way the current assumptions around ~/.julia/bin hold but we also get the benefits of using the XDG-appropriate dir as outlined in my first comment.

A related question, according to XDG where should the .julia/environments/apps/Package folder go?

I made a flowchart for answering this sort of question in the BaseDirs.jl docs which might be helpful (it's not 100% accurate, but I didn't want to make it more complicated, and I think it gets 98% of the way).

If we classify .julia/environments/apps/Package as:

• user-specific data
• not something explicitly configured by the user
• unable to be deleted without causing disruption to the system behaviour

then Data Home would be the relevant XDG Desktop component (let me know if any of those assumptions don't hold).

More generally, I find .julia/environments/ a bit interesting in that it's a mix of automatically-changed and user-modified environments. The v1.x environments are changed when the user explicitly asks for a package to be installed/removed, and so line up best as "user configuration". However, you also have environments like __pluto_boot_v2_1.8.5 which are very much not, and probably best classed as user data.

For Windows the Cargo issue comment says:

For Windows, everything should go in ~/appdata/locallow or ~/appdata/local, since ~/.cargo is just a cache, AFAICT. This is FOLDERID_LocalAppData for SHGetKnownFolderPath, CSIDL_LOCAL_APPDATA for SHGetFolderPath, and %LOCALAPPDATA% in the environment.

How is that translated to all the files used here (shims, AppManifest.toml, app environments)?

A while ago I spent an inordinate amount of time looking at the relevant behaviour/specs/comments around directories on Windows/Mac. I think I'd probably be best off pointing you to the comparison table on https://tecosaur.github.io/BaseDirs.jl/stable/defaults/ (and if you want the reasoning/links to some of the most relevant resources: https://tecosaur.github.io/BaseDirs.jl/stable/others/).

Regarding just this part of the comment:

This is FOLDERID_LocalAppData for SHGetKnownFolderPath, CSIDL_LOCAL_APPDATA for SHGetFolderPath, and %LOCALAPPDATA% in the environment.

Yea, getting the right system dirs on windows is actually a bit of a pain. See https://github.com/tecosaur/BaseDirs.jl/blob/main/src/nt.jl for a glimpse of me not having a fun time.

[davidanthoff]

one plan forward is to use Juliaup to install the Julia installation that the app is currently configured for if it does not exist. That way you would not store the absolute path to the julia installation like that.

My plan generally is that the Julia version in a manifest becomes the version selector for Juliaup. Presumably that would work well for apps here too?

[ufechner7]

What is still needed before this can be merged?