Cannot save multi-line output in a variable (support $() command substitution syntax)

[lilyball]

There seems to be no way to capture multi-line output in a shell variable. Any attempt to do so splits each line into a separate array element. In bash I'd simply put double-quotes around my $() invocation, but in fish you cannot quote a command substitution. This is rather unfortunate because it means I cannot capture multi-line output in a variable and send it back to a separate command without really weird contrivances, such as replacing all newlines with NUL when saving and reverting the process when emitting.

[lilyball]

And for the record, doing such a replacement makes it impossible to look at the status of the original command* (e.g. in issue #158), because Fish provides no replacement for bash's PIPESTATUS.

*without really stupid things like piping the output to a file temporarily

[lilyball]

As a separate idea, if there was some flag to echo that made it emit each argument on a separate line instead of space-separation, that would at least provide a workaround. And yes, this is something that can be shellscripted. But it really shouldn't be necessary.

[lilyball]

The lack of a way to quote a command substitution has just reared its head in another location: the inability to use command substitution with test -n. If the command substitution results in no output, it also results in no arguments at all, and test -n with no subsequent arguments actually returns true.

[maxfl]

For me it looks quite natural: no output - no arguments. One empty line of output - one empty argument.
What is strange for me is why 'test -n' returns 0 at all? Naively I would suppose it to return 1.
I would suggest to change the returning value for this case, but the system 'test' and the bash internal 'test' also return 0 in this case.

Concerning the multi-line output I completely agree with you. It would be very nice to have possibility to do command substitution inside double quotes, something like "$()", because substituting "()" is dangerous.

[lilyball]

As an alternative to $() (which, btw, I still want just so I can more easily combine command substitution with surrounding variables, etc), it might be nice to just have a trivial way of taking a multi-element variable and creating one argument with all elements joined by a newline. This would effectively reverse the process of storing a command substitution into a variable, with the one exception that this will force a trailing newline when the command substitution may not have had one. Right now I can write (for elem in $var; echo $elem; end), but that's rather awkward to do everywhere.

[maxfl]

Shorter ways to do the same:
(printf '%s\n' $var)
or
(echo $var\n)?
But fish again splits the substitution into arguments by newlines. How do you use it?

[lilyball]

I assume you mean printf, and that would work but it has to spawn a separate process which is slow.

As for (echo $var\n), that's going to put a space after every newline, which is wrong, as well as include two newlines at the end.

The use of this is for piping to another process. Obviously storing it back in a variable would just split it again, but I can say (for line in $var; echo $line; end | grep foo)

[maxfl]

You are right.
echo you can use when you do not care about extraspace (like in grep example). And printf when you do not care about preformance.

[ridiculousfish]

This outputs 1:

count (printf "%s %s %s" first second third)

This outputs 3:

count (printf "%s\n%s\n%s" first second third)

I don't understand why newlines from a substitution result in arrays, while spaces from a substitution do not. I wonder how bad it would be to treat whitespace uniformly here.

[maxfl]

I think that it's perfectly consistent behaviour (considering that fish distinguishes whitespaces). From my experience automatic expanding space-separated strings to array makes it difficult to cope with space containing strings and lead to rather extensive subquoting.

From the other hand: additional splitting can be implemented by additional command. But if fish will split all whitespaces it will be difficult to glue them back if needed.

[ridiculousfish]

My suggestion was actually in the other direction: treat newlines like spaces (so they do not split). So

count (printf "%s\n%s\n%s" first second third)

would output 1. On the other hand, then we would need some way to split a string.

I also think kballard's suggestion of quoted substitutions via e.g. "$(echo hello)" is very good and would address this neatly.

[maxfl]

I completely agree about "$(echo hello)".
I also sometimes think, that replacing () by $() in normal mode is a good
way, because in this case there will be no need to escape ().

[Soares]

Personally I'm very -1 to $() syntax.

One of the things that drew me to fish was that it simplifies all of the crazy different ways to do things that bash/zsh/etc have.

See the design doc, the law of orthogonality. To quote,

The shell language should have a small set of orthogonal features. Any situation where two features are related but not identical, one of them should be removed
...

• The many Posix quoting styles are silly, especially $''.

I think it would be a serious degradation of fish's principles to add $() syntax.

From what I understand, adding a third quoting syntax ("" vs '' and unquoted) was a large and difficult decision. If we're going to do a change like this we should at least strongly consider all alternatives first.

Understanding the whitespace rules right now is a big pain point. Currently:

> count 'one two three'
1
> count one two three
3

That I get. Quotes prevent splitting. Cool.

> count (echo one two three)
1

Ok and it looks like subcommands are automatically considered quoted...

> count one\ntwo\nthree
1

And newlines don't count as whitespace when you're splitting things, that's cool I guess

> count (echo one\ntwo\nthree)
3

Wait, what?! Something's seriously smelly here. If (echo one two three) acts like 'one two three' then (echo one\ntwo\nthree) should act like 'one\ntwo\nthree'.

And, of course, the big scripting problem is that the data isn't preserved by set:

> set foo (echo one\ntwo)
> echo $foo
one two

We had newlines and now we have spaces.

I understand why it all works like this but it definitely doesn't seem very fish-like. User focus is violated here, orthogonality certainly isn't helped. I'm not sure what the answer is, but any or all of these are options I like better than $():

1. Substitution output should be treated exactly like a quoted string. If count "one\ntwo\nthree" is 1 then count (echo one\ntwo\nthree) should be 1
2. Alternatively, make substitution output be unquoted and allow it inside double quotes. This is a pretty big semantic change.
3. Don't split output on newlines by default. Add a dice command which splits output-in-lines into quoted-args-separated-by-spaces. This is analogous to how we have psub instead of process substitution: fish should prefer new commands to new syntax.
4. Give set a new flag which tell it how to dice variables.
   (Pros: you could split things on colons if that's your thing. Con: doesn't help count or () usage.)
5. Give echo a new flag which tells it how to separate args (fixes the symptoms not the problem.)

Concerning option 2, it would mean that unquoted command substitution would act like this:

> count one two three
3
> count "one two three"
1
> count (echo one two three)
3
> count "(echo one two three)"
1

Which is a big change, but is at least consistent.

Many of these are more work-arounds than solutions. And even if we fix set and echo by adding more flags we still have the count elephant in the room, which can't take any flags of any sort: if we add flags to set that tell it how to split arrays then it seems pretty silly that count can never have the same functionality.

I'm most in favor of 1. or 2. above, but either way I'm against adding $() syntax. One more design doc quote:

Most tradeoffs between power and ease of use can be avoided with careful design.

Most of my ideas above are still half-baked, but I think more carefully designing the text-splitting rules is a far superior option to adding more syntax and flags on top of the existing ones in an attempt to patch all the holes. That sort of activity is what got bash where it is today.

[ridiculousfish]

Thank you for the thoughtful comments Soares.

I think there might be a point of confusion. The $() proposal is to add a way to do command substitutions within double quotes. It need not work outside of them:

> count (echo 1\n2\n3)
3
> count "$(echo 1\n2\n3\n)"
1
> count $(echo 1\n2\n3\n)
syntax error

In that way I think it's similar to your suggestion 2 above. The dollar sign has the advantages of not requiring escaping parens within double quotes (which would be irritating), and because it doesn't conflict with any existing valid syntax.

This was an element of kballard's "Fish Scripting Wishlist" from June 21. To quote Kevin:

1. Fish needs a way to do command substitution within a double-quoted string. The simplest solution is probably to support $() within double-quotes. The reasoning is twofold: first, a command substitution that evaluates to no output ends up being stripped entirely from the argument list of its surrounding command. This has extremely bad implications when using it with, e.g. test -n (some command). Second, this would allow for storing the output with its newlines in a variable, instead of having the output be split into multiple elements. And as a bonus it would make it a lot easier to combine command substitution output with other text in the same argument, e.g. "$somevar"(some command)"$anothervar".

So "$()" would solve several problems, which is why it's interesting.

I think we'd also like to avoid splitting on newlines as you suggest in 3 above. The main blocker there is the sheer quantity of work required to vet all existing fish code, and in adding the new 'dice' command.

[Soares]

Cool. I'm much less opposed to that syntax if it's only in double quotes.

(Note: In fish 1.x variables in double quotes expanded to the first arg in that variable array and there was no way to slice/get the other args. This was a concession to Axel Liljencrantz IIRC, who didn't want double quotes in the first place and insisted on having something that set double quotes apart, which I thought was stupid. I was right on the brink of a rant about the magical things that double quotes do before I realized that double quotes are pretty sane in fish2.0. Nice work on that!)

It's still weird to me to use $() instead of (). I understand how much of a pain in the ass it would be to escape all parens in double quotes, but isn't that sort of the point of single quotes instead of double quotes?

Also I'm wary of conflating $() in double quotes with POSIX $(); it could be confusing to newcomers. I'd prefer #() or {} syntax to distance ourselves a bit from the dollar sign, which so far only means variable expansion. (The fact that it's not used in () sets something of a precedent for not re-using the dollar sign.)

Also I think that expanding an empty variable should yield '', because even if we do have expansion in double quotes you still have the old

> set var
> test -n $var

problem, which is a serious gotcha for newcomers.

[lilyball]

What is POSIX $()? Bash has been supporting $() in double-quotes to do exactly what we're suggesting here for quite a long time, and nobody has a problem with that. Adding yet another syntax for process expansion seems like a really really bad idea. You say dollar sign only means variable expansion so far. That's fine, but I see no problem with expanding that to just meaning expansion in general, with the two supported expansion types being either variables or subprocesses.

Expanding an empty variable should never yield '' unless that variable is enclosed in double-quotes. Changing that would basically introduce a requirement to use eval whenever you want to conditionally add an argument, which is a really bad idea. I wouldn't worry too much about test -n $var; newcomers who don't understand what they're doing have worse issues than eliding an argument due to empty variable expansion.

[Soares]

Bash's $() is what I meant when I said posix (is that posix?), as I didn't want to single out bash. Bash supports $() syntax everywhere, not just in double quotes: if fish supports it but only in special circumstances that seems a bit weird.

Especially since fish keeps command substitution but changes the syntax from $() to (). Switching from $() to () for command substitution implies that fish is trying to distance itself from the ``` $() ${} <()hellhole that is bash/zsh substitution; it seems weird to turn around and put$()` syntax back in double quotes.

Which is why I'm tentatively in favor of biting the bullet and allowing () in double quotes, though that's pretty backwards incompatible.

[lilyball]

I would consider interpreting () inside double-quotes as substitution to be extremely surprising and absolutely terrible. Currently the only substitution that occurs inside of double-quotes (note: escaping is not substitution) requires a $ character. There is no good reason to introduce any separate ways to invoke substitution inside of double-quoted strings.

[Soares]

Yeah, I concede that that might be the best course of action from where we're standing. However, the converse to your statement is this:

Remember that the only substitution that occurs outside of quotes is $ and () characters. There is no good reason to add new syntax for the sole purpose of restricting one of the existing substitution methods.

Having $ and () substitute outside of quotes while having $ and $() substitute inside of quotes is messy and dichotomous and exactly the sort of thing that fish (as upposed to bash/zsh/etc) is supposed to avoid. It's in direct opposition to a few parts of the design doc and it raises the question of "if I use $() in a string why don't I use it outside of a string?".

It's a shit situation. () was chosen for command substitution back when fish didn't have double quoted strings. $var is already supported in double quoted strings, so we can't do it 'ruby-style' and say that #{} un-double-quotes you, so that you have to do variables like #{$var} and command substitutions like #{(command args)}.

[lilyball]

If you really want to be pure, you could introduce $() outside of double-quoted strings as well, and deprecate the bare () style. This way "the only substitution that occurs requires $" will be true regardless of quoting.

Of course, at this point, you may then say "what about {}?" While not strictly speaking substitution, you could then argue that we should change that to ${} as well. And actually, I'd have no objection to that. The fact that fish interprets {} specially even when there's only one branch inside is very annoying when trying to work with git. Plus you could then allow ${} inside of double-quoted strings without a problem.

That said, from a practical standpoint, I think making these changes here is unnecessary. I'd rather just introduce $() inside of double-quoted strings and be done with it.

[Soares]

Yeah, if the language were being designed from the get-go I'd recommend axing the bare version. If $() is added to double quoted strings then I definitely think there should be some sort of long-term plan to restore consistency, which I think is important to fish in its role as a bastion against shell scripting insanity.

Just a note: If you have long double quoted strings, you can already

> "have command substitution "(in double-quoted strings)" like this."

It's one character more and it doesn't require extra syntax.

It's really only the edge-case of "$(some expression)" where the new syntax is useful, and I'd prefer to see that fixed by sane newline handling and a nice dice function. If we get those I'm not sure we even need the $() syntax.

[lilyball]

No you can't.

> echo "test"(echo one\ntwo)"ing"
testoneing testtwoing

The desired behavior is

>echo "test$(echo one\ntwo)ing"
testone
twoing

[Soares]

Right. Sorry. I meant to say that you can do that if we make variables not be split on newlines by default and add a dice command (as discussed above), then the existing syntax works.

[ridiculousfish]

Soares, out of curiosity, where did you get name for the dice command? We're interested in adding some more sophisticated string manipulation, and I'd rather adopt an existing syntax than invent a new one.

[Soares]

For some reason I thought the existing split command was named "slice". Given that split splits things by length and not content I thought a command that split things by content and not length could be called dice (as in 'slice and dice'.)

Seeing now that I was thinking of split, slice is probably a much better name for the command we're discussing.

[maxfl]

Soares, thank you for that long argumented answer. I now agree with you (:

[gustafj]

@Soares, very well written and good arguments, I agree with consistency here, I don't see the point of having one syntax in double quotes and one outside of double quotes.
I also wouldn't want to type $() for every command substitution (outside of "") I do daily, but I see the problem of changing the meaning of () inside "".
@kballard, the issue with {} is solved by #354 (remove it).

To summarize, either only $var & $() or $var & (), I would prefer the second option, although it breaks horribly.

[lordlycastle]

Last related comment was over a year ago. This is still a big missing feature with no good/standard way of doing it. Are we not happy with "$(echo $var)"? Are we waiting for something that can be helped?

[lilyball]

We have string collect now, but as I stated before I think we need to add a discovery mechanism for users before we can close this issue.

[ridiculousfish]

I agree with keeping this open. We'll probably end up supporting "$(cmd)" syntax, but it's a lot of work.

First step I guess is to design it. In bash "$(cmd)" establishes a new quoting context:

echo "$(echo "nested quotes!")"

do we want to allow that, to arbitrary depth? Or can we start with something useful but more restricted, which would be easier to implement?

[krobelus]

I think we can hack the parser to close double quotes on $( and reopen them after the matching).
So echo "b$(...)c" is virtually tokenized as echo "b"$(...)"c".
This way we get nesting, syntax highlighting and completion for free - see my implementation.

The cost here is the more complex quoting syntax - a single regex will no longer be enough to correctly highlight such strings.
Probably there are ready solutions to support this kind of syntax in most 3rd party highlighters.

If we don't support quotes inside "$()", then the highlighting syntax would remain unchanged.
The nesting is less important than in bash but it feels natural to include it eventually.

Not sure what to do about unquoted $(), I feel like it should mean the same as "$()" but previous suggestions use the () meaning. We can also leave it unsupported for now.

---

Alternatively, if we want to avoid making the quoting syntax more complex, we could support unquoted $() only :

echo "no interpolation "$(echo "(but nested quotes)")" for you"

That may be less elegant but does not require to understand the concept of command substitutions inside quotes.
In situations where no quotes are required for the surrounding text (like the OP), it's also more concise.

We could try to make this as convenient to type as the bash version:
when the user types "foo$(, fish could just insert "foo"$( instead...

[pirate]

The new support for $(cmd)/"$(cmd)" added in #8059 is amazing, thank you so much @krobelus for working on this, and @ridiculousfish for approving it!! 🎉 👏 👏 👏 ✨

---

For anyone curious on how the new "$(cmd)" support works, the documentation for it is added here: bf5d2d1 (#8059).

Usage Documentation

A command substitution can have a dollar sign before the opening parenthesis like outercommand $(innercommand). This variant is also allowed inside double quotes. When using double quotes, the command output is not split up by lines.

A command like echo (cmd | string collect) is mostly equivalent to a quoted command substitution (echo "$(cmd)"). The main difference is that the former evaluates to zero or one elements whereas the quoted command substitution always evaluates to one element due to string interpolation.

fish's command substitution syntax has been extended: $(cmd) now has the same meaning as (cmd) but it can be used inside double quotes, to prevent line splitting of the results. (:issue:159).

Example Usage

$ echo "zero $(echo one\ntwo\nthree) four"
    zero one
    two
    three four

---

And just to close things out I want to quote my favorite comment #8059 (comment) from the PR by @ridiculousfish themself regarding @krobelus's implementation 😁 👏 :

ridiculousfish: @krobelus I stand in awe. My approach was to teach fish about cmdsubs in quotes everywhere, which meant a lot of complexity and recursion. Your trick of making cmdsubs "temporarily close" quotes is so brilliantly simple, I would not have believed it possible! I learned from it today.

---

P.S. @krobelus do you have a Paypal/Patreon/Github Sponsors/GitTip/etc.? I'd love to send you a tip as a thank you for working on this feature and knocking it out of the park, your work here closes one of the longest outstanding issues I've been waiting for on the fish project, and I'd love to show my thanks. 🙏