[script] recipes

[casey]

Existing recipes, both shebang and linewise, have some issues:

The issues:

• The just parser looks for {{…}} interpolations inside of recipe bodies. This requires escaping {{ inside recipe bodies, which is awkward.
• You can't set variables in the middle of a linewise recipe. This requires using shebang recipes. Having two types of recipes is confusing.
• If interpolated variables and arguments inside of {{…}} contain quotes, spaces, or other special characters, quoting is quite tricky.
• The difference between environment variables and just variables is confusing.
• Shebang lines are interpreted differently by different OSes. On Unix-like OSes, they're interpreted by the kernel, and different kernels split them into arguments differently. On Windows, shebang scripts are unsupported, so Just implements its own shebang line parsing.
• Having two kinds of recipes to choose between is confusing. It would be better if the default recipe was useful for all use-cases.
• In linewise recipes, \ must be used to escape multi-line constructs
• Positional arguments are useful for avoiding needing to quote arguments, but must be opted into using a setting

The proposal:

• A new recipe type which is indicated with a [script] annotation
• Scripts are saved to a file and executed with sh PATH_TO_SCRIPT, and do not take shebang lines
• The script interpreter can be customized, either with an annotation on specific recipes[script("bash", "-i")] or a global setting that changes the default script interpreter
• The just parser does not look for {{…}} inside of script bodies
• set positional-arguments is always turned on for scripts, so arguments are accessed with $1, $2, etc
• Since interpolations are turned off, just variables must be exported to be visible to scripts

Downsides:

• When executing a linewise recipe, just echoes each recipe line before running it. This would not be possible for scripts. However, a script could use set -x to turn this on. Additionally, there could be a global script prefix, which could default to something like set -euxo pipefail.
• Since the just parser wouldn't process interpolations in script recipes, variable name typos wouldn't be caught statically
• Adding a third recipe type would be confusing
• The need to export environment variables to script recipes would pollute downstream processes. However, in practice I don't think this is really a problem.

Eventually, [script] recipes could be made the default recipe type in a future just edition, and linewise and shebang recipes could be deprecated. The migration process would look like this:

1. Add [script] recipes behind the --unstable flag
2. Stabilize [script] recipes once they're working well
3. Add an annotation for non-script recipes, like [legacy], for the current default recipe type
4. Create a new edition which makes [script] recipes the default

Migration would be easy. Users could add the [legacy] annotation to their recipes, update to the new edition, and convert [legacy] recipes to [script] recipes at their leisure.

I'm very interested in feedback on this!

[indigoviolet]

Is this a common tripping hazard for new users? TBH, this doesn't resonate with me as a a big pain point that is worth the loss of functionality (no interpolation, no echoing, have to remember to export variables) with not much new functionality enabled, complexity added in the interim:

• interpolation of {{ }} and escaping: I do use this interpolation more often than I escape it. Perhaps the flag we're looking for is no-interpolation instead

• shebang line/interpreter customization -- has not been a problem for me (but I don't use just on windows)

• set positional-arguments: I haven't understood why this isn't on by default - is there a downside?

I already avoid shebang recipes as much as possible: (1) too many footguns in shell scripts (2) justfiles are not good development environments for complex scripts. I can already write a shell script and execute it from just if needed.

My suggestion would be to instead aim to not send users to shebang/script recipes. In that context, I would like to make it easier to treat other recipes in the file as functions -- not necessary syntactically, but in the sense of making it easy to compose complex recipes from simpler ones; there are some awkward edges like requiring -f {{ justfile() }}, constructing arguments by string concatenation, conditionals being expressions.

[casey]

I should go back and look at issues, discussions, and discord chat, but just going on memory, there are some common tripping hazards that this would avoid:

• Very common: Confusion about how to prevent {{…}} interpolations from being split by the shell. Needing to use '{{…}}', which is pretty ugly and verbose, doesn't work if an argument has ' in it.
• Very common: Desire to set variables in the middle of a recipe.
• Occasional: Shebang inconsistencies.

set positional-arguments isn't the default just because it can break some custom shells (namely old versions of powershell), but it's likely to become the default in a new edition.

It would be interesting to see how common shebang recipes are. One could use janus to find out.

Can you elaborate on the downsides? You mentioned loss of interpolation, echoing, and exporting variables:

• Loss of interpolation: {{FOO}} would be replaced with $FOO (or $N if FOO is an argument), which is shorter and is the normal shell syntax.
• Echoing: just could define a [script] recipe prelude which would be inserted at the beginning of script recipes. This could include set -x, which would turn on command echoing.
• Exporting variables: In a future edition, set export could become the default, so you wouldn't have to remember to export variables.

[casey]

By the way, thanks for the detailed feedback, I really appreciate it!

[indigoviolet]

1. Interpolation: I understand that with positional-arguments and export, we could get rid of interpolation for variables; what about for functions or expressions? ex {{ invocation_directory() }}?

Going in the other direction, I miss interpolation outside of recipes, concatenation is more verbose (#11). If this feature got added, we'd have the inverse confusion - why isn't {{..}} working in my recipe?

2. Desire to set variables

I'm not understanding how [script] will address this trip hazard: it doesn't make it easier or eliminate the confusion between just variables and shell variables.

[casey]

Going in the other direction, I miss interpolation outside of recipes, concatenation is more verbose (#11). If this feature got added, we'd have the inverse confusion - why isn't {{..}} working in my recipe?

Interesting point. I'm not sure if users would want {{…}} inside recipes, since the shell usually provides functionality to manipulate strings.

I'm not understanding how [script] will address this trip hazard: it doesn't make it easier or eliminate the confusion between just variables and shell variables.

I wasn't very clear, but I mentioned this because it's one of the main reasons that users need shebang recipes.

[nk9]

I'd just like to point out that the current setup allows interpolation in all recipes, so the syntax is able to deal with that very simply. If it were possible for some recipes to disallow interpolation, it wouldn't be at all straightforward for the syntax to distinguish those. We'd have to branch at the point of the [script] attribute and create a category of non-interpolation recipes which duplicate the recipe code. I'm not sure it would be worth the effort.

I'm not taking a position on whether the attribute is worthwhile, but if there's not a clear case for it then my preference would be to not add it.

[alexxbb]

I'm looking forward to seeing this implemented, this is my biggest hurdle with just right now.

registry_main := "registry_main URL"

publish reg:
    ./scripts/publish.py {{ if reg == "main" { "{{registry_main}}" } else { "{{reg}}" } }}

>>just publish main
>>./scripts/publish.py {{registry_main}}

[casey]

I'm now leaning towards not doing this, and instead adding a [script] recipe annotation which is simply a way to create a shebang recipe with cross-platform syntax, see #1479.