Make it possible to pass boolean flags dynamically

[Veetaha]

Related problem

Suppose we have a command that wraps another command where we want to forward --push and --release boolean flags to the inner command called build-imp.

def build-imp [--push, --release] {
    # This isn't a real example. Here we would actually do smth useful like building a docker image
    # optionally in release mode and optionally pushing it to the remote registry
    echo $"Doing a build with push: ($push) and release: ($release)"
}

At the time of this writing, it isn't possible to do this naturally. The following doesn't work, because nu complains that there are excessive positional parameters $push and $release:

def build [--push, --release] {
    build-imp --push $push --release $release
}

Error message

Error: nu::parser::extra_positional (link)

  × Extra positional argument.
   ╭─[entry #6:1:1]
 1 │ def build [--push, --release] {
 2 │     build-imp --push $push --release $release
   ·                      ──┬──
   ·                        ╰── extra positional argument
 3 │ }
   ╰────
  help: Usage: build-imp {flags}

This also doesn't work, although it compiles. Any invocation of build (even with all flags unset) results in $push and $release arguments inside build-imp set to true.

def build [--push, --release] {
    build-imp --push=$push --release=$release
}

Demo of the variant higher

❯ build
Doing a build with push: true and release: true
❯ build --push
Doing a build with push: true and release: true
❯ build --push --release
Doing a build with push: true and release: true
❯ build --release
Doing a build with push: true and release: true

Workarounds

Combinatorial explosion

Handle all possible combinations of all flags with an if-else chain. This solution preserves the desired command signature but requires a huge amount of boilerplate. This solution suffers from copy-paste exponentially (pow(2, number_of_flags)) with the total number of flags to forward. The situation is worsened if there are other arguments to pass other than the flags that will also need to be copy-pasted

def build [--push, --release] {
    if $push and $release {
        build-imp --push --release
    } else if $push and not $release {
        build-imp --push
    } else if not $push and $release {
        build-imp        --release
    } else {
        build-imp
    }
}

A variation of the solution that uses a bit nicer grouping of combinations with bitflags

def build [--push, --release] {
    let invocations = [
        { build-imp                  }
        { build-imp           --push }
        { build-imp --release        }
        { build-imp --release --push }
    ]

    do ($invocations | get (($release | into int) bit-shl 1 bit-or ($push | into int)))
}

Use non-boolean type for flags

UPD: there was found a better version of this workaround that allows you to use booleans, but doesn't enable a bare --flag syntax: #7260 (comment)

The downside of this approach is that we make the API uglier, although the boilerplate doesn't grow exponentially here. We can define the underlying command with int arguments, but also make a wrapper that provides the same nice boolean flag API and just converts its arguments to integers.

def build-imp-int [--push: int, --release: int] {
    let $push = $push == 1
    let $release = $release == 1
    # ...
}

# If this function needs to be reused in a bunch of places where boolean flags API suffices,
# then we may define a simple wrapper around it that takes boolean flags and forwards
# them to the inner function with `int` flags, but that's also boilerplate, so not ideal:
def build-imp [--push, --release] {
    build-imp-int --push ($push | into int) --release ($release | into int)
}

Describe the solution you'd like

The easiest solution that may probably be implemented quickly is to allow using --flag=boolean_expression syntax for boolean flags. The equals sign (=) between the argument name and the boolean expression disambiguates that the argument following the boolean flag isn't a positional parameter.

Right now we can pass literally any expression using --flag=expression, but that results in the boolean $flag variable to be set to true unconditionally. Maybe this was the intended solution of this problem at all, but it doesn't work today. Maybe it's a bug then?

Describe alternatives you've considered

A more general solution to the problem may also encompass solving the problem of passing optional flags to external commands.

For example, today we can't do this:

let push = if condition { "--push" } else { null }

docker buildx build $push ...args...

Because nutshell will complain that it can't convert null to string. If we were to use an empty string instead of null nushell would forward it to the external binary as a separate empty string positional parameter. So instead we have to do this today:

let args = [...args...]
let args = if condition { $args | prepend "--push" } else { $args }

docker buildx build $args

This works because an array is spread to positional parameters when invoking an external binary from nushell, but it also looks like some kind of inconvenient dark magic.

A more general problem is omitting named parameters to commands. I think it deserves a special syntax like this:

docker buildx build (--push $push)? (--named-arg-with-value $value)?

With the question mark operator appended nushell would omit the parameter if the value is false or null for example. This is quite a naive first-comes-to-mind solution. I bet there could be some limitations with this, so feel free to share better proposals.

[melMass]

Do you have other use cases for flags that accepts booleans or is it just to solve passing them to external commands?
The alternatives part I believe this is doable with run-external, I haven't tried myself but I had the exact same case to solve a few weeks ago and someone mentioned run-external on discord recently I think it fits this usecase.

[Veetaha]

I don't think run-external improves the experience for external commands. How would one use it to pass the --push flag based on the boolean value stored in $push variable? I don't see other approaches than specified in alternatives (i.e. make the array of args and extend it dynamically).

[melMass]

Maybe I was not clear but I meant that this shoud be doable in run-external

def do_it [--push] {
	let flags = []

	let flags = if ($push) {
		$flags | append "--push"
	} else {
		$flags
	}
	run-external "echo" $flags
}

Obviously not ideal I agree

[melMass]

Oh my bad, while first reading the issue I skipped exactly this part of your issue (where you suggest the same solution without run-external)!

[Veetaha]

@melMass

Regarding the uses case for non-external commands. I have a development script that has commands:

• "main docker build" [--push, --release] - build the app's docker image
• "main deploy" [--release] - deploy the app and automatically build it; it has to inherit the same --release flag as build command

I need to forward the release flag from deploy command into build command, and now it looks quite clumsy due to the limitation described in this issue.

[theAkito]

Because nutshell will complain that it can't convert null to string. If we were to use an empty string instead of null nushell would forward it to the external binary as a separate empty string positional parameter.

Related to #7033 due to quote & the whole issue in its entirety.

This whole forced to String conversion problem is a huge super-topic. That it is really hard to dynamically generate command chains is one of its biggest effects.

[amtoine]

@Veetaha
does the following solve your issue? 😋

def build-imp [--push: bool = false, --release: bool = false] {
    echo $"Doing a build with push: ($push) and release: ($release)"
}

def build [--push, --release] {
    build-imp --push $push --release $release
}

which gives

>_ build
Doing a build with push: false and release: false
>_ build --push
Doing a build with push: true and release: false
>_ build --release
Doing a build with push: false and release: true
>_ build --push --release
Doing a build with push: true and release: true

[cumber]

@amtoine Honestly I kind of expected something like def build [--push] to be syntactic sugar for def build [--push : bool = false], and build --push to be syntactic sugar for build --push=true. It's really surprising that build --push=false neither passes $push as false nor throws a syntax error!

[amtoine]

the build --push=false part looks a bit odd 🤔

however, the rest looks sensible to me

def foo [--bar] {
    if $bar {
        print "bar!"
    } else {
        print "no bar..."
    }
}

does give

> foo
no bar...

> foo --bar
bar!

which is expected, isn't it? 😋

[melMass]

I think the OP wants to conditionally set flags for a wrapper. Hence the need for a value flag (--b:bool). If I understand right this is to wrap externals.
Although there is no equal in nu arg so you must do --b true

Afaik there is no fix for all that yet

[Veetaha]

The suggestion from @amtoine regarding the def build-imp [--push: bool = false, --release: bool = false] slightly improves the situation, but then you can no longer call the build-imp this way:

build-imp --push --release

You can only call it this way now

build-imp --push true --release false

[amtoine]

The suggestion from @amtoine regarding the def build-imp [--push: bool = false, --release: bool = false] slightly improves the situation, but then you can no longer call the build-imp this way ...

this is very true @Veetaha 👍

i use this trick inside modules, when the build-imp-like command wouldn't leak to the outside, which is fine when it does not impact the user API imo 😋

but yeah for interactive usage, that might not be ideal 😕

[crabdancing]

I ran into this too! Glad to see there's already an issue on it. This isn't a general solution, but it's what I came up with to soften the pain of this:

export def "append-if" [cond: bool, target: any] -> list {
    if $cond {
        $in | append $target
    } else {
        $in
    }
}

# List installed unit files
export def "sctl list unit-files" [
    --all: bool # List all unit files
    --user: bool # Run as non-root
] {
    let args = ['-o' 'json'] | append-if $all "--all" | append-if $user "--user"
    systemctl list-unit-files $args | from json
}

Unfortunately, it doesn't work on Nushell's custom commands

[crabdancing]

I think if you make a flag for a builtin, it should be possible to transparently handle it the following ways:

def my_cmd[--flag: bool] {
}

1. Iff this flag is of type bool, it will be false if absent, by default, and true if present. This is consistent with a lot of shell standards, and is fairly intuitive.

2. Iff this flag is of type bool, you can also assign values via --flag=<value>

It's noteworthy that this could create surprising behavior if the type is not explicit -- the user could be trying to pass in a string, forget the string, and then suddenly somehow it's a bool. :P

To avoid that, it might be best if this only happens if you explicitly type the argument.

Side note:

This proposed solution does not fix some of the other potential pains of working with this system -- for example, wrapping Nushell custom commands with other Nushell custom commands. You'd still end up having to do a lot of boilerplate, if there's a lot of flags. Whatever solution we adopt, I hope it'll be future-proof against addressing this use case as well. My suggestion for that would be some kind of splat syntax. If it's specific to Nushell custom commands, it could also include other desirable behaviors, like copying the signature information from the child function to the parent -- though I'm not sure how to implement that in a fully-general way.

[vector-of-bool]

For what its worth, the syntactic sugar suggested above has precedent in PowerShell, which is enabled by a distinct "switch" parameter type:

param (
    # A boolean parameter, which must be given a value
    [bool]
    $Meow,
    # A switch, which looks like a bool for most intents and purposes
    [switch]
    $Woof,
    # A positional
    [string]
    $Greeting
)

Some-Function -Meow $true -Woof 'hello'

The way pwsh handles the above invocation depends on the types of the parameters. Since -Meow has type bool, it will bind to the next expression. -Woof has type switch, so it does not grab the next expression, allowing 'hello' to bind to the next unbound positional parameter -Greeting implicitly. Explicitly binding a switch uses the colon:

Some-Function -Woof:$some_boolean

I would imagine introducing a new similar explicit switch annotation could be a backwards-compatible way to introduce such a feature, in the case that users may be (unintentionally) relying on the existing behavior.

This would have the unfortunate effect of complicating parsing, depending on how/when commands are parsed. i.e. One cannot fully interpret the meaning of foo --bar $boolean_value without knowing whether --bar is a switch or a bool. Determining whether $boolean_value is being given as a positional or as a named argument for --bar requires peering into the declaration of foo.

The underlying task of "include a string in an exec() array depending on a condition" is something I stub my toe on constantly in every programming language and library where I need to invoke external processes, and really boils down to this very problem: the overwhelming majority of external programs do not provide a way to explicitly set the value of a toggle other than the presence/absence of a specific token in the argv array. Having Nu break this troublesome Unix-ish mold would be a breath of fresh air 🙂.

[cumber]

I think that's overcomplicated. We already have the syntax, it just needs to be made to work.

❯ def foo [--flag, x : int] { echo { flag: $flag, x: $x} }

❯ foo 2
╭──────┬───────╮
│ flag │ false │
│ x    │ 2     │
╰──────┴───────╯

❯ foo --flag=false 3
╭──────┬──────╮
│ flag │ true │
│ x    │ 3    │
╰──────┴──────╯

Nushell's parser is able to recognise that the string --flag=false is trying to provide a value for flag (as it's true in the output; when it doesn't think any value is being set then it defaults to false). It doesn't think that it's setting a flag named flag=false or anything; it is already able to correctly split it up into the flag name flag before the = and a value afterwards. However the current behaviour for boolean options is to simply ignore the value, and set the value of the flag to true if any option with that name was passed.

We don't need a distinction between booleans and switches; we just need the --var=value syntax that works for every other type to also work for booleans. The only thing that needs to be special about boolean arguments is that they don't require an explicit value, and they should never consume the following word (an explicit value can only be provided with the = syntax).

In my opinion flags should just be very minor surface-level syntactic sugar on top of the same logic that works for named options of any type. --flag should just be shorthand for --flag=true (when $flag is known to be of boolean type).

[vector-of-bool]

Your response led me down a rabbit hole.

I had a misunderstanding in that I believed the following were different:

def f [--x] {}
def g [--y: bool] {}

My intuition was that --y is a named parameter that accepts a bool value, while x is a switch that sets $x=true. My intuition was that:

def h [--z: T] {}

meant "declare --z as a named parameter that accepts a value of type T". This appears to be untrue if T is bool. It appears that bool type annotations are intentionally discarded by the parser 😢.

That got me curious to try this:

def m [--u: bool = "Hello"] {
  print "Hello"
}

which checks and executes, while giving --u a type string, and not bool. (But this behavior changed just a few hours ago.)

I then went on:

def foo [--xyz: any = 42] {}
foo --xyz "string"

Until recently, this code would fail, as the parser would override the any annotation with int. This issue was known (#10410) and was coincidentally fixed three hours ago in #10424.

IIUC: Whether a flag consumes a value is determined by its syntax_shape. Initially, all flags are given None syntax shape, which causes the "--switch" parsing behavior. The reason that --some-toggle=false sets $some_toggle=true is because the --some-toggle has a syntax_shape of None, and there is no type validation on the =<value> part of the flag if syntax_shape is None.

The syntax shape of a flag is set when the parser encounters the type or when it encounters a default value. Despite not changing the type, a bool type is still distinct from providing an explicit any, because the code that discards the bool also skips setting a syntax_shape, which is why the --foo=$boolean syntax simply sets $foo=true, while setting a type any will set the syntax_shape, and cause $foo=$boolean.

The reason [--foo: bool = false] works to create a named boolean is that: because setting the argument's type is skipped, this guard is never matched, and it will then override the syntax shape with the inferred bool from the default value.

However, as of the recent changes in #10424, an additional guard was added, and I hypothesize that [--foo: bool = false] will no longer work as noted above, since A) the type of --foo is still any, and B) arg_explicit_type is now true, therefore skipping overriding the argument type and overriding the syntax shape (I do not have a recent build to test this hypothesis).

However ×2, this still leaves [--foo = false], which will infer a named argument of type bool while arg_explicit_type == false, and therefore will still override the syntax shape, created a value-consuming flag of type bool.

In summary:

# Creates an value-less switch "--xyz", with $xyz: bool
def foo [--xyz] {}

# Same. (IMO, an unintuitive special case for only `bool`)
def foo [--xyz: bool] {}

# Creates a named flag "--xyz=..." which expects a `bool`
def foo [--xyz = false] {}

# As of yesterday: Same as above (broken)
# As of #10424:    Creates a named flag "--xyz" which takes any value
def foo [--xyz: any = false] {}

# As of yesterday: Creates a named flag "--xyz=..." which expects a `string` (broken)
# As of #10424:    Creates a named flag "--xyz=..." which expects any value (should probably be an error)
def foo [--xyz: bool = "hey"] {}

---

Regardless, there's still a question of whether back-compat is significant in this weird case. i.e. if def m [--xyz] {}; m --xyz=false were "fixed" to produce $xyz=false. The existing (surprising) behavior of $xyz being true is unintuitive, but there's a non-zero chance that someone somewhere is relying on this behavior, even if unknowingly (obligatory).

[cumber]

Is there a separate "bug" with syntax_shape of None accepting but ignoring any value? Surely it should be an error to provide an option value when none can be accepted, instead of silently discarding it.

(If that had been done from the start, then --xyz=false would have been an error, and there would not now be any backwards compatibility concerns about adding a new feature to allow explicitly providing an argument for boolean arguments)

Or am I misunderstanding and syntax_shape None is used in some cases when an argument is expected?

It seems that the original/current behaviour of boolean flags should have a syntax shape that can never accept arguments. And to resolve this enhancement issue we just need a new syntax shape that accepts an optional value (which presumably means the argument can only be provided by the --option=value syntax; permitting --option value would be ambiguous about whether value is a value for --option or a separate positional argument).

[flying-sheep]

I had a misunderstanding in that I believed the following were different:

def f [--x] {}
def g [--y: bool] {}

My intuition was that --y is a named parameter that accepts a bool value, while x is a switch that sets $x=true. My intuition was that:

As of nu 0.86.0, those are different in exactly the way you intuited. FWIW, I had the opposite intuition. Type annotations shouldn’t change semantics.

Anyway, the changes in #10456 didn’t help with the present issue. As @amtoine said in #10456 (comment):

the general rule of thumb is never define --switch: bool in the public API 👍

So yeah, you can now define options that can explicitly be passed a boolean value for your private APIs. But for public APIs and 3rd party commands, one should still do --xy/--no-xy, not --xy=true/--xy=false. So we still need an ergonomic way to do that.

And IMHO once we have that, the changes in #10456 are nothing but an unnecessary second way to do it that has no reason for existing anymore.

[crabdancing]

I was thinking we need a way of concisely expressing that something is true (--xy) while retaining the ability to negate that value explicitly, and ideally without having to add any boilerplate on the user side of things. I think if you add an --xy as a boolean flag, you should be able to pass it alone to enable it, but also optionally add --xy=false to disable it. Or at least, that's the best I've come up with so far. The downside is, I don't know how well it fits with the rest of the syntactic standards.

[theAkito]

I was thinking we need a way of concisely expressing that something is true (--xy) while retaining the ability to negate that value explicitly, and ideally without having to add any boilerplate on the user side of things. I think if you add an --xy as a boolean flag, you should be able to pass it alone to enable it, but also optionally add --xy=false to disable it.

Precisely. That is exactly, what is needed. It's intuitive.

[cumber]

I was thinking we need a way of concisely expressing that something is true (--xy) while retaining the ability to negate that value explicitly, and ideally without having to add any boilerplate on the user side of things. I think if you add an --xy as a boolean flag, you should be able to pass it alone to enable it, but also optionally add --xy=false to disable it. Or at least, that's the best I've come up with so far. The downside is, I don't know how well it fits with the rest of the syntactic standards.

Agreed, that's exactly what I proposed in this thread a while back. I'm actually a little disappointed we've moved in the direction of differentiating between "switches" and arguments that happen to have type bool, instead of tweaking the syntactic sugar so that boolean arguments can be passed explicitly.

The parser even already accepted --xy=false, it's just that the behaviour of that syntax was obviously incorrect (set xy to true); fixing that bug was all that was required here.

[crabdancing]

@cumber Yeah, that's exactly how I feel too! It seems very intuitive to have absence be false by default, presence being true by default, and = syntax to explicitly set values for shell scripting. As it is now, it's a lot more verbose than it has to be; and, if you have like 8 different boolean flags, you're going to have to type a lot more, which is a problem for interactive use cases! Plus, it reduces congruence with other tooling.

[cumber]

@cumber Yeah, that's exactly how I feel too! It seems very intuitive to have absence be false by default, presence being true by default, and = syntax to explicitly set values for shell scripting.

And we can't really evolve this new non-switch boolean argument in that direction, because it (uniformly with other types) accepts a space-separated value as in --flag false. If a bare --flag is a way of specifying that you're passing true for flag, then --flag false is ambiguous about whether it's setting flag to false, or setting flag to true (by simple presence) followed by an unrelated positional argument that is false.

So once there's any usage of these new boolean arguments out there it would be backwards incompatible to start accepting bare --flag syntax as a user convenience, and backwards incompatible to remove either style of boolean flag/switch in favour of the other.

[crabdancing]

Is the reason this was done purely a matter of syntactic uniformity? It seems the community is leaning in the direction of special behavior for boolean flags, anyway. (Which could be later made uniform by having flags accept either --my-flag my-value or --my-flag=my-value.) The ecosystem is small enough still that a breaking change isn't that big a deal (At least, IMO!) and is worth setting a standard that will cause the least sadness. :P

[flying-sheep]

I think if you add an --xy as a boolean flag, you should be able to pass it alone to enable it, but also optionally add --xy=false to disable it. Or at least, that's the best I've come up with so far. The downside is, I don't know how well it fits with the rest of the syntactic standards.

That would be a great solution if we lived in a vacuum (i.e. it was a great solution for React when they were free to design JSX however they liked: <Comp prop /> is the same as <Comp prop={true} />, so you can pass a variable when deriving this prop value and have concise syntax when not.)

But as I said above: It doesn’t help in the slightest with 3rd party commands. There I still have to do stuff like

let args = ([(if ($info) { ['-i'] } else []), $pkg] | flatten)
^my-cmd $args

[theAkito]

But as I said above: It doesn’t help in the slightest with 3rd party commands. There I still have to do stuff like

Already pointed as well, that this is actually a bigger, essential issue, which crosses a lot of Nushell surface.

Related to #7033 due to quote & the whole issue in its entirety.

This whole forced to String conversion problem is a huge super-topic. That it is really hard to dynamically generate command chains is one of its biggest effect

This specific super-issue is already tracked in #7033.

[WindSoilder]

Thanks for all your comments! Let me try to summarize the path we're trying to move:

switch

For the given example:

def foo [--xyz] { print $xyz }

It needs to support: foo(print false), foo --xyz(print true), foo --xyz=false(print false), foo --xyz=true(print true), and we can only set it with true or false, something like this will result to error: foo --xyz=1

It doesn't support: foo --xyz false, foo --xyz true (For the reason please check the comment: #7260 (comment))

flag which takes boolean value

For the given example:

def foo2 [--xyz: bool] { print $xyz }
def foo2 [--xyz = false] { print $xyz }

I'd like to make it as normal as other flags which takes a value for consistency (I have the same institution to #7260 (comment)). Which is handled by #10456

[theAkito]

It needs to support: foo --xyz, foo --xyz=false, foo --xyz=true, and we can only set it with true or false, something like this will result to error: foo --xyz=1

The default will be set to false though, right?

[WindSoilder]

Yup, if we call foo directly, the value of $xyz is false, I've updated the comment.

[cumber]

Thanks @WindSoilder, that sounds exactly right.

It needs to support: foo(print false), foo --xyz(print true), foo --xyz=false(print false), foo --xyz=true(print true)

I assume you intended this, but just to be explicit: it should also support foo --xyz=<EXPR>, where <EXPR> stands for any expression that could be used as a flag argument (provided it results in a Boolean value of course). Most usefully the expression could be a variable reference, e.g. foo --xyz=$abc.