Allow --project=@scriptdir for easier co-packaging of scripts and p…

[Keno]

…rojects

I have a directory with a number of runnable scripts and a Project.toml that applies to all of them. I was hoping to have a simple shebang line for these, but there didn't really seem to be a good way to set the --project without forcing the script to be invoked with a specific cwd or playing complicated shebang games. With this PR, you can simply write:

#!/usr/bin/env -S julia --project=@scriptdir/..

and everything will work out of the box. Perhaps we already have a good mechanism for this, but if so, I was unable to find it, so in that case, consider this a documentation issue instead ;).

[KristofferC]

The CLI docs calls the "script" argument as "programfile". So should it be "programdir"?

[Keno]

I don't really care about the name, but a little worried that programdir would get interpreted as the dir of julia.

[KristofferC]

The feature itself seems reasonable to me. Most scripts resort to using Pkg.activate(@__DIR__) which is not great since it requires Pkg.

[vtjnash]

SGTM. I guess the only issue is that GNU only added this -S feature in 2018, so it didn't show up in ubuntu LTS until 2020. (the BSD's have had it since circa 2005 with FreeBSD 5.5, although OpenBSD and busybox do not implement it).

Random thought: could we potentially detect if the interpreter and script (argv[0]) are the same, and operate in a script mode? Maybe this won't be reliable though, since I think uv_exepath is itself potentially derived from argv[0])

julia> p = zeros(UInt8, 256); len = Ref{Csize_t}(sizeof(p)); ccall(:uv_exepath, Cint, (Ptr{UInt8}, Ref{Csize_t}), p, len); String(resize!(p, len[]))
"/Users/jameson/julia/usr/bin/julia"

Or could we simply make something like this the default when handling a given script as the argument, unless specified otherwise?

[Keno]

Or could we simply make something like this the default when handling a given script as the argument, unless specified otherwise?

We could, but I don't think it's clear that we always know which is the correct Project to use. We could potentially consider doing our own shebang parsing so you could do something like:

#!/usr/bin/env julia
#! --project=@scriptdir --check-bounds=no

but that seems orthogonal. I also think it's fine to rely on -S. Can always compile the busybox version of env if necessary.

[vtjnash]

Busybox in particular does not have options for env except for the posix ones: https://busybox.net/downloads/BusyBox.html

[KristofferC]

Or could we simply make something like this the default when handling a given script as the argument, unless specified otherwise?

The reason this is not done is AFAIU to prevent a malicious Manifest file that's been accidentally downloaded into e.g. Downloads to be able to change what julia Downloads/foo.jl executes.

[Keno]

Seems like people generally agree that this could be reasonably useful, so I'm just gonna get this in so people can play with it. If people want a different name, we can still change it until the 1.11 release.

[ericphanson]

I think this should be documented and get a NEWS entry. Currently on 1.9 this syntax activates a shared environment named @scriptdir:

❯ julia --project=@scriptdir
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@scriptdir) pkg> st
Status `~/.julia/environments/scriptdir/Project.toml` (empty project)

That seems somewhat of an edge case but a NEWS entry would probably at least help alert folks to this change if they happen to be using a shared environment with that name.

[digital-carver]

Currently on 1.9 this syntax activates a shared environment named @scriptdir

While this is unlikely to break anyone's workflow in practice, it would be nice if we systematized this right away and had a clear distinction between "This refers to a shared environment" vs "This refers to special keyword (like scriptdir) that has specific behaviour".

Some options are:

• @@scriptdir - easy to type and remember
• @~scriptdir - mirrors the way some shells allow to designate specials directory bookmarks (eg. zsh)
• @$scriptdir - mirrors the Julia interpolation syntax

Adopting any one of these will help both in making it highly unlikely that these special options clash with existing shared environments, and with making it clear immediately at a glance that this is a custom value being given to the option.

[vtjnash]

We have an convention already of @ being special, with users expected to write ./<name> if you want a local path. Some existing examples include:

"@", "@v#.#", "@stdlib", "@." are valid in LOAD_PATH & friends
--track-allocation=@<path>
Profile.print abbreviates paths with `@<Pkg>/`

[digital-carver]

Yes, @xyz is already special as a value to --project too - it starts Julia with the shared environment xyz activated. The current PR overrides that for specifically when xyz is scriptdir, which makes it a special case within a special case.

That's why I'm suggesting that it should have unique syntax indicating that something new is going on here, and not just a usual shared environment activation.

[tecosaur]

I like this idea! A little thought though, what about instead of doing @scriptdir/.. etc. we just do @script and have it act like @. in the script directory? To me, the benefit of it being shorter is that it's even more trivial to remember (it's a minor thing, but I find that when there are two-word special names to remember and I haven't used them in a while, I tend to forget the order etc.), and I can't myself think of a situation where anything more than acting like @. would be wanted.

[mkitti]

What happens if I have a global shared environment called "scriptdir". Normally, I would access it via julia --project=@scriptdir".

[mkitti]

I like passing arguments via the initial comment block. There is precedent for this in SLURM.

https://slurm.schedmd.com/sbatch.html

sbatch submits a batch script to Slurm. The batch script may be given to sbatch through a file name on the command line, or if no file name is specified, sbatch will read in a script from standard input. The batch script may contain options preceded with "#SBATCH" before any executable commands in the script. sbatch will stop processing further #SBATCH directives once the first non-comment non-whitespace line has been reached in the script.