Skip to content

Cannot save multi-line output in a variable #159

Open
kballard opened this Issue Jun 20, 2012 · 45 comments
@kballard

There seems to be no way to capture multi-line output in a shell variable. Any attempt to do so splits each line into a separate array element. In bash I'd simply put double-quotes around my $() invocation, but in fish you cannot quote a command substitution. This is rather unfortunate because it means I cannot capture multi-line output in a variable and send it back to a separate command without really weird contrivances, such as replacing all newlines with NUL when saving and reverting the process when emitting.

@kballard

And for the record, doing such a replacement makes it impossible to look at the status of the original command* (e.g. in issue #158), because Fish provides no replacement for bash's PIPESTATUS.

*without really stupid things like piping the output to a file temporarily

@kballard

As a separate idea, if there was some flag to echo that made it emit each argument on a separate line instead of space-separation, that would at least provide a workaround. And yes, this is something that can be shellscripted. But it really shouldn't be necessary.

@kballard

The lack of a way to quote a command substitution has just reared its head in another location: the inability to use command substitution with test -n. If the command substitution results in no output, it also results in no arguments at all, and test -n with no subsequent arguments actually returns true.

@maxfl
maxfl commented Jun 20, 2012

For me it looks quite natural: no output - no arguments. One empty line of output - one empty argument.
What is strange for me is why 'test -n' returns 0 at all? Naively I would suppose it to return 1.
I would suggest to change the returning value for this case, but the system 'test' and the bash internal 'test' also return 0 in this case.

Concerning the multi-line output I completely agree with you. It would be very nice to have possibility to do command substitution inside double quotes, something like "$()", because substituting "()" is dangerous.

@kballard

As an alternative to $() (which, btw, I still want just so I can more easily combine command substitution with surrounding variables, etc), it might be nice to just have a trivial way of taking a multi-element variable and creating one argument with all elements joined by a newline. This would effectively reverse the process of storing a command substitution into a variable, with the one exception that this will force a trailing newline when the command substitution may not have had one. Right now I can write (for elem in $var; echo $elem; end), but that's rather awkward to do everywhere.

@maxfl
maxfl commented Jun 20, 2012

Shorter ways to do the same:
(printf '%s\n' $var)
or
(echo $var\n)?
But fish again splits the substitution into arguments by newlines. How do you use it?

@kballard

I assume you mean printf, and that would work but it has to spawn a separate process which is slow.

As for (echo $var\n), that's going to put a space after every newline, which is wrong, as well as include two newlines at the end.

The use of this is for piping to another process. Obviously storing it back in a variable would just split it again, but I can say (for line in $var; echo $line; end | grep foo)

@maxfl
maxfl commented Jun 20, 2012

You are right.
echo you can use when you do not care about extraspace (like in grep example). And printf when you do not care about preformance.

@ridiculousfish
The user-friendly shell member

This outputs 1:

count (printf "%s %s %s" first second third)

This outputs 3:

count (printf "%s\n%s\n%s" first second third)

I don't understand why newlines from a substitution result in arrays, while spaces from a substitution do not. I wonder how bad it would be to treat whitespace uniformly here.

@maxfl
maxfl commented Oct 9, 2012

I think that it's perfectly consistent behaviour (considering that fish distinguishes whitespaces). From my experience automatic expanding space-separated strings to array makes it difficult to cope with space containing strings and lead to rather extensive subquoting.

From the other hand: additional splitting can be implemented by additional command. But if fish will split all whitespaces it will be difficult to glue them back if needed.

@ridiculousfish
The user-friendly shell member

My suggestion was actually in the other direction: treat newlines like spaces (so they do not split). So

count (printf "%s\n%s\n%s" first second third)

would output 1. On the other hand, then we would need some way to split a string.

I also think kballard's suggestion of quoted substitutions via e.g. "$(echo hello)" is very good and would address this neatly.

@maxfl
maxfl commented Oct 10, 2012
@Soares
Soares commented Oct 17, 2012

Personally I'm very -1 to $() syntax.

One of the things that drew me to fish was that it simplifies all of the crazy different ways to do things that bash/zsh/etc have.

See the design doc, the law of orthogonality. To quote,

The shell language should have a small set of orthogonal features. Any situation where two features are related but not identical, one of them should be removed
...

  • The many Posix quoting styles are silly, especially $''.

I think it would be a serious degradation of fish's principles to add $() syntax.

From what I understand, adding a third quoting syntax ("" vs '' and unquoted) was a large and difficult decision. If we're going to do a change like this we should at least strongly consider all alternatives first.

Understanding the whitespace rules right now is a big pain point. Currently:

> count 'one two three'
1
> count one two three
3

That I get. Quotes prevent splitting. Cool.

> count (echo one two three)
1

Ok and it looks like subcommands are automatically considered quoted...

> count one\ntwo\nthree
1

And newlines don't count as whitespace when you're splitting things, that's cool I guess

> count (echo one\ntwo\nthree)
3

Wait, what?! Something's seriously smelly here. If (echo one two three) acts like 'one two three' then (echo one\ntwo\nthree) should act like 'one\ntwo\nthree'.

And, of course, the big scripting problem is that the data isn't preserved by set:

> set foo (echo one\ntwo)
> echo $foo
one two

We had newlines and now we have spaces.

I understand why it all works like this but it definitely doesn't seem very fish-like. User focus is violated here, orthogonality certainly isn't helped. I'm not sure what the answer is, but any or all of these are options I like better than $():

  1. Substitution output should be treated exactly like a quoted string. If count "one\ntwo\nthree" is 1 then count (echo one\ntwo\nthree) should be 1
  2. Alternatively, make substitution output be unquoted and allow it inside double quotes. This is a pretty big semantic change.
  3. Don't split output on newlines by default. Add a dice command which splits output-in-lines into quoted-args-separated-by-spaces. This is analogous to how we have psub instead of process substitution: fish should prefer new commands to new syntax.
  4. Give set a new flag which tell it how to dice variables. (Pros: you could split things on colons if that's your thing. Con: doesn't help count or () usage.)
  5. Give echo a new flag which tells it how to separate args (fixes the symptoms not the problem.)

Concerning option 2, it would mean that unquoted command substitution would act like this:

> count one two three
3
> count "one two three"
1
> count (echo one two three)
3
> count "(echo one two three)"
1

Which is a big change, but is at least consistent.

Many of these are more work-arounds than solutions. And even if we fix set and echo by adding more flags we still have the count elephant in the room, which can't take any flags of any sort: if we add flags to set that tell it how to split arrays then it seems pretty silly that count can never have the same functionality.

I'm most in favor of 1. or 2. above, but either way I'm against adding $() syntax. One more design doc quote:

Most tradeoffs between power and ease of use can be avoided with careful design.

Most of my ideas above are still half-baked, but I think more carefully designing the text-splitting rules is a far superior option to adding more syntax and flags on top of the existing ones in an attempt to patch all the holes. That sort of activity is what got bash where it is today.

@ridiculousfish
The user-friendly shell member

Thank you for the thoughtful comments Soares.

I think there might be a point of confusion. The $() proposal is to add a way to do command substitutions within double quotes. It need not work outside of them:

> count (echo 1\n2\n3)
3
> count "$(echo 1\n2\n3\n)"
1
> count $(echo 1\n2\n3\n)
syntax error

In that way I think it's similar to your suggestion 2 above. The dollar sign has the advantages of not requiring escaping parens within double quotes (which would be irritating), and because it doesn't conflict with any existing valid syntax.

This was an element of kballard's "Fish Scripting Wishlist" from June 21. To quote Kevin:

  1. Fish needs a way to do command substitution within a double-quoted string. The simplest solution is probably to support $() within double-quotes. The reasoning is twofold: first, a command substitution that evaluates to no output ends up being stripped entirely from the argument list of its surrounding command. This has extremely bad implications when using it with, e.g. test -n (some command). Second, this would allow for storing the output with its newlines in a variable, instead of having the output be split into multiple elements. And as a bonus it would make it a lot easier to combine command substitution output with other text in the same argument, e.g. "$somevar"(some command)"$anothervar".

So "$()" would solve several problems, which is why it's interesting.

I think we'd also like to avoid splitting on newlines as you suggest in 3 above. The main blocker there is the sheer quantity of work required to vet all existing fish code, and in adding the new 'dice' command.

@Soares
Soares commented Oct 17, 2012

Cool. I'm much less opposed to that syntax if it's only in double quotes.

(Note: In fish 1.x variables in double quotes expanded to the first arg in that variable array and there was no way to slice/get the other args. This was a concession to Axel Liljencrantz IIRC, who didn't want double quotes in the first place and insisted on having something that set double quotes apart, which I thought was stupid. I was right on the brink of a rant about the magical things that double quotes do before I realized that double quotes are pretty sane in fish2.0. Nice work on that!)

It's still weird to me to use $() instead of (). I understand how much of a pain in the ass it would be to escape all parens in double quotes, but isn't that sort of the point of single quotes instead of double quotes?

Also I'm wary of conflating $() in double quotes with POSIX $(); it could be confusing to newcomers. I'd prefer #() or {} syntax to distance ourselves a bit from the dollar sign, which so far only means variable expansion. (The fact that it's not used in () sets something of a precedent for not re-using the dollar sign.)

Also I think that expanding an empty variable should yield '', because even if we do have expansion in double quotes you still have the old

> set var
> test -n $var

problem, which is a serious gotcha for newcomers.

@kballard

What is POSIX $()? Bash has been supporting $() in double-quotes to do exactly what we're suggesting here for quite a long time, and nobody has a problem with that. Adding yet another syntax for process expansion seems like a really really bad idea. You say dollar sign only means variable expansion so far. That's fine, but I see no problem with expanding that to just meaning expansion in general, with the two supported expansion types being either variables or subprocesses.

Expanding an empty variable should never yield '' unless that variable is enclosed in double-quotes. Changing that would basically introduce a requirement to use eval whenever you want to conditionally add an argument, which is a really bad idea. I wouldn't worry too much about test -n $var; newcomers who don't understand what they're doing have worse issues than eliding an argument due to empty variable expansion.

@Soares
Soares commented Oct 17, 2012

Bash's $() is what I meant when I said posix (is that posix?), as I didn't want to single out bash. Bash supports $() syntax everywhere, not just in double quotes: if fish supports it but only in special circumstances that seems a bit weird.

Especially since fish keeps command substitution but changes the syntax from $() to (). Switching from $() to () for command substitution implies that fish is trying to distance itself from the `` $() ${} <() hellhole that is bash/zsh substitution; it seems weird to turn around and put $() syntax back in double quotes.

Which is why I'm tentatively in favor of biting the bullet and allowing () in double quotes, though that's pretty backwards incompatible.

@kballard

I would consider interpreting () inside double-quotes as substitution to be extremely surprising and absolutely terrible. Currently the only substitution that occurs inside of double-quotes (note: escaping is not substitution) requires a $ character. There is no good reason to introduce any separate ways to invoke substitution inside of double-quoted strings.

@Soares
Soares commented Oct 17, 2012

Yeah, I concede that that might be the best course of action from where we're standing. However, the converse to your statement is this:

Remember that the only substitution that occurs outside of quotes is $ and () characters. There is no good reason to add new syntax for the sole purpose of restricting one of the existing substitution methods.

Having $ and () substitute outside of quotes while having $ and $() substitute inside of quotes is messy and dichotomous and exactly the sort of thing that fish (as upposed to bash/zsh/etc) is supposed to avoid. It's in direct opposition to a few parts of the design doc and it raises the question of "if I use $() in a string why don't I use it outside of a string?".

It's a shit situation. () was chosen for command substitution back when fish didn't have double quoted strings. $var is already supported in double quoted strings, so we can't do it 'ruby-style' and say that #{} un-double-quotes you, so that you have to do variables like #{$var} and command substitutions like #{(command args)}.

@kballard

If you really want to be pure, you could introduce $() outside of double-quoted strings as well, and deprecate the bare () style. This way "the only substitution that occurs requires $" will be true regardless of quoting.

Of course, at this point, you may then say "what about {}?" While not strictly speaking substitution, you could then argue that we should change that to ${} as well. And actually, I'd have no objection to that. The fact that fish interprets {} specially even when there's only one branch inside is very annoying when trying to work with git. Plus you could then allow ${} inside of double-quoted strings without a problem.

That said, from a practical standpoint, I think making these changes here is unnecessary. I'd rather just introduce $() inside of double-quoted strings and be done with it.

@Soares
Soares commented Oct 17, 2012

Yeah, if the language were being designed from the get-go I'd recommend axing the bare version. If $() is added to double quoted strings then I definitely think there should be some sort of long-term plan to restore consistency, which I think is important to fish in its role as a bastion against shell scripting insanity.

Just a note: If you have long double quoted strings, you can already

> "have command substitution "(in double-quoted strings)" like this."

It's one character more and it doesn't require extra syntax.

It's really only the edge-case of "$(some expression)" where the new syntax is useful, and I'd prefer to see that fixed by sane newline handling and a nice dice function. If we get those I'm not sure we even need the $() syntax.

@kballard

No you can't.

> echo "test"(echo one\ntwo)"ing"
testoneing testtwoing

The desired behavior is

>echo "test$(echo one\ntwo)ing"
testone
twoing
@Soares
Soares commented Oct 17, 2012

Right. Sorry. I meant to say that you can do that if we make variables not be split on newlines by default and add a dice command (as discussed above), then the existing syntax works.

@ridiculousfish
The user-friendly shell member

Soares, out of curiosity, where did you get name for the dice command? We're interested in adding some more sophisticated string manipulation, and I'd rather adopt an existing syntax than invent a new one.

@Soares
Soares commented Oct 17, 2012

For some reason I thought the existing split command was named "slice". Given that split splits things by length and not content I thought a command that split things by content and not length could be called dice (as in 'slice and dice'.)

Seeing now that I was thinking of split, slice is probably a much better name for the command we're discussing.

@maxfl
maxfl commented Oct 18, 2012

Soares, thank you for that long argumented answer. I now agree with you (:

@gustafj
gustafj commented Oct 23, 2012

@Soares, very well written and good arguments, I agree with consistency here, I don't see the point of having one syntax in double quotes and one outside of double quotes.
I also wouldn't want to type $() for every command substitution (outside of "") I do daily, but I see the problem of changing the meaning of () inside "".
@kballard, the issue with {} is solved by #354 (remove it).

To summarize, either only $var & $() or $var & (), I would prefer the second option, although it breaks horribly.

@pgan002
pgan002 commented Dec 13, 2012

How about an option 2a: Substitution output should be treated like an unquoted string, and instead of dice or slice and list, splitting and joining can be done using the IFS variable (as discussed for Fish 1.0). Something like:

> count a b
2
> count (echo a b)
2  # As we usually want
> set -l IFS ''; count (echo a b)
1
> set -l IFS '\n'; count (echo a b)
1
@kballard

I hate IFS. IMO it's one of the worst parts of bash scripting.

@Soares
@JanKanis
The user-friendly shell member

Just now reading up on this due to the list discussion.

One thing missing in this discussion is why splitting behavior currently is as it is:

me@mypc ~/test> ls -1
another file
file 1
third file
me@mypc ~/test> count (ls)
3

If proposal 1 were implemented:

me@mypc ~/test> count (ls)
1

if proposal 2 or 3 were implemented:

me@mypc ~/test> count (ls)
6

Newline and space/tab in current unix behave as a level 1 and level 2 separator. That comes (I guess) from human text where newline and space in a sense do the same. In command output, neither of these already mean something and sometimes you need spaces, so the logical choice was to use the level 1 separator. When entering commands, the newline is already taken as separating different commands, so the level 2 separator was used. This is how all command line unix software got written, and fish just follows by converting the level 1 separator used in command output to the level 2 separator that count or any command line argument uses. There are zillions of programs that give output in one line per item format.

This allows e.g. using of spaces in filenames. Of course it breaks when there's a newline in a filename or anywhere where you don't want it to split output, and the general design turned out to suck, but you can't change it without changing all command line unix software ever written! Doing so would make fish suck for interacting with non-fish-aware commands -- and therefore as a shell.

So -1000 on changing the default whitespace splitting. That said, I'm in favor of something like a list command, and I also hate $IFS.

@dag
dag commented May 10, 2013

I really think consistency is king and fish should do one of:

  1. Allow all substitutions and escapes and expansions inside double quotes, with the exact same syntax. Quotes then simply delimit arguments containing whitespace, without changing any semantics. If you want a literal string either escape the special characters or use single quotes.
  2. Remove support for double quotes or make them behave like single quotes. Quotes then are "string literals".

For option 1 I'm strongly against changing the syntax for command substitution, and for option 2 we need some other way to say "treat this command substitution as a single string regardless of newlines". I favor the first option.

@dag
dag commented May 10, 2013

Or 3. have an "unquote" syntax like Ruby's #{} but I'd prefer it to be just {}. I'm not a fan of this option as "{(ls)}" is a bit awkward and it means double quotes are almost the same thing as single quotes.

@Undeterminant

The first option for that sounds perfectly valid and it'd probably solve a lot of problems. It seems more orthogonal.

@kevna
kevna commented Dec 4, 2013

I don't know about others here but I rarely use literal () characters as part of a string, if other people feel this way then perhaps it would not be so significant to have to escape them to prevent substitution.

Alternatively, are there any side issues to the following - use ( to begin a substitution ( to use brackets normally (or vice versa) where one is followed by a space.

@xfix
The user-friendly shell member
xfix commented Dec 4, 2013

@kevna: I like that one. I wasn't sure about suggesting it, but now I think it's a good idea. When I type ( anyway, it's part of code (for example Perl oneliner) or regular expression, and usually I write code in single quotes to avoid variables being written into result.

Then again, some code (mostly related to completions uses ( in double quotes). But I think that completions are relatively easy to fix.

@Undeterminant

I dunno how much of this has actually been worked on, but I just thought of a better alternative: why not just make (( and )) literally paste the output of the command as one argument? It looks similar to the bash syntax for a conditional, but in cases where bash conditionals would be used, it would throw an error for invalid syntax anyway. (just having a subshell as the command doesn't work)

@xixixao
xixixao commented Dec 21, 2013

+1 to any solution to this!

@xixixao
xixixao commented Mar 7, 2014

That isn't multiline.

@xfix
The user-friendly shell member
xfix commented Mar 7, 2014

I made this just for fun. The real solution would be nice, obviously, as currently there is no real way to do so. This is a really needed feature, the issue is that nobody can decide on its syntax. Obviously, pipeset is not a proposed syntax, it's just a function because I cannot create new syntax from the fish shell itself.

function pipeset --no-scope-shadowing
    set -l _options
    set -l _variables
    for _item in $argv
        switch $_item
            case '-*'
                set _options $_options $_item
            case '*'
                set _variables $_variables  $_item
        end
    end
    for _variable in $_variables
        set $_variable ""
    end
    while read _line
        for _variable in $_variables
            set $_options $_variable $$_variable$_line\n
        end
    end
    return 0
end

Example:

~ $ perl --version | pipeset perl_version
~ $ echo $perl_version

This is perl 5, version 14, subversion 4 (v5.14.4) built for cygwin-thread-multi
(with 7 registered patches, see perl -V for more detail)

Copyright 1987-2013, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.


~ $
@kaleb
kaleb commented Mar 7, 2014

@xfix That's a nice solution. It would be nice if one could just pipe to set.

> perl --version | set perl_version
> echo $perl_version
@anordal
anordal commented Mar 7, 2014

PHP's explode/implode functions look easy to adopt to a shell syntax:
http://php.net/manual/en/function.explode.php
http://php.net/manual/en/function.implode.php
(IMO, more straightforward & recognizable than python's split/join)

I am very much in favor of ridiculousfish's "no treatment of whitespace" in variables (should be compatible with dag's suggestion 1, if I get him right, as he's talking about quoting whitespace itself). Having to quote variables to keep them together (keep them from "exploding", so to say) is the number one thing I hate about posix shells.** Explosion should be explicit!

In other words, no implicit array interpretation, so

count (echo a\nb)

would output 1, whereas if I actually want to do that kind of string processing,

count (explode \n (echo a\nb))     #Look ma, no IFS

would output 2.
As can be guessed, there is a reverse command, implode. This would output 1:

count (implode \n (explode \n (echo a\nb)))

As a bonus, we then get some string processing primitives!
Search & replace:

implode 'replace' (explode 'search' $mystring)

Basename:

(explode / $mypath)[-1]

** What posix shells really really want to do, is to explode your variables, eat your data as commands, and kill you! How many times, in writing sizable shell scripts, have you stopped to think how many bugs you're just creating, and contemplated a safer programming language like C?

@xixixao
xixixao commented Apr 8, 2014

@xfix Thanks for your code, works like a charm!

@ShadowKyogre ShadowKyogre added a commit to ShadowKyogre/dotfiles that referenced this issue Sep 29, 2014
@ShadowKyogre ShadowKyogre Try to implement explode and implode because fish-shell/fish-shell#159.
Still not sure how to explode strings without sed.
b2ea87f
@derekstavis

Another great syntax that runs away from $ is what Swift proposes for string interpolation:

set CPUINFO "\(cat /proc/cpuinfo)"
@jakwings

I'm opposed to ((...)), #{...}, {...}, "\(...)". And "$(...)" is the best since $ is already a special symbol for variables. If we also change the syntax of unquoted command substitution to $(...), that would be better. Then we can use () in test (...) -a (...) without escaping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.