Syntax improvements for redirection & pipes

[dhasenan]

Assuming I have a process that writes to both stderr and stdout, and I want to pipe both to tee, I should be able to do:

./a.out ^&1 | tee log

This ends up with 'log' being empty.

I suspect that fish is assigning the stderr input stream to the stdout sink, then assigning the stdout input stream to the pipe sink. Another layer of indirection would likely solve this -- instead of copying the stdout sink, you assign a wrapper sink instead. This would require caution to catch the case of redirecting >&2 ^&1, of course, but it would allow people to redirect and pipe at the same time and make it work.

[dag]

I ran into this issue almost immediately so I'd love to see a solution.

[JanKanis]

Can anyone give or link to an exact description of how bash handles the interaction of redirections and pipes? I can't find it in the bash reference manual. I always thought the bash equivalent of the o.p.'s command would give the same result as fish, but it turns out that command 2>&1 | tee myfile sends both stdout and stder of command to tee.

My (apparently wrong) mental model of redirection in bash is that cmd1 2>&1 | cmd2 does:

• assign the value of fd 1 (the terminal) to cmd1's fd2
• assign the value of the pipe to cmd1's fd1.
  Fish (afaik) works like that, but what does bash do exactly?

[gustafj]

In the manual for Bash under Pipelines i found this:
This connection is performed before any redirections specified by the command.
see http://www.gnu.org/software/bash/manual/bashref.html#Pipelines

So the redirection would be like this if my understanding is correct:

• connect the fd1 to the pipe
• assign the value of fd1 (the pipe) to cmd1's fd2

It is possible to do this today in fish using special syntax 2>| ex command 2>| tee myfile
In Bash this is equivalent to |& which is just syntactic sugar for 2>&1 |

I did a few trials using fish, bash and tcsh (note that my tcsh dont support to only redirect stderr, as noted in that cmd):

cmd is perl -e 'print {*STDOUT} "stdout\n"; print {*STDERR} "stderr\n";' in all below:

tcsh > cmd | cat - > /dev/null
stderr
tcsh > cmd |& cat - > /dev/null
tcsh > cmd |& cat -
stderr
stdout
tcsh > "The  shell  cannot  presently  redirect  diagnostic  output without also redirecting standard output"

bash-4.1$ cmd | cat - > /dev/null
stderr
bash-4.1$ cmd |& cat - > /dev/null
bash-4.1$ cmd |& cat -
stderr
stdout
bash-4.1$ cmd 2>&1 | cat - > /dev/null


fish > cmd | cat - > /dev/null
stderr
fish > cmd 2>| cat - > /dev/null
fish > cmd 2>| cat -
stdout
stderr
fish > cmd 2>&1 | cat - > /dev/null 
stderr

[dag]

My expectation would be that 2>| would pipe stderr but not stdout. And also that if that syntax works then so should ^|, doing the same. Piping both is likely more useful, but might arguably break "The law of user focus" in that it's not intuitive.

[gustafj]

I like the ^| syntax, feels more "fish-like", and i also agree that when using 2>| (or ^|) one could expect it to only pipe stderr.
But then what syntax would be appropriate to use when piping both (which is probably what you most often want, except for when only piping stdout).

Proposal:

• Only stdout |, >| or 1>|
• Only stderr ^| or 2>|
• Both stderr & stdout ^>| or >^| or any combination of redirection followed by pipe, ex: ^&1 | or >&2 ^|

[dag]

The good thing is that the syntax is beginning to look like a fish ...

[ridiculousfish]

Let’s try to fix this one in next-minor

[ridiculousfish]

If you can pipe stdout and stderr independently, how do pipes associate? For example:

foo ^| bar | baz

Does baz get the output of foo or bar? Anyone know how other shells handle this?

[timbertson]

If we had subshells, I assume the answer would be that you'd need to do the following if you want stderr to go through multiple processes:

$ foo ^| (bar | baz) | frob

stderr goes through baz | bar, while stdout goes through frob. I thought the above would be possible in bash, but it seems you only get one pipe in bash (http://stackoverflow.com/questions/2342826/how-to-pipe-stderr-and-not-stdout)

Is there any subshell-like thing in fish? I often want to do stuff in another dir temporarily, which I used to do in bash with (cd DIR; do-something). I miss that.

[KamilaBorowska]

@gfxmonk: But we do.

foo ^| begin
    bar | baz
end | frob

[anddam]

@glitchmr that's brilliant

[ridiculousfish]

It sounds like the proposed answer to my question is that a stderr-only pipe would be standalone, and not part of the larger pipeline. That is, foo ^| bar | baz is foo ^| bar and foo | baz. If you wanted to pipe stderr further, you use begin/end to create a block. That seems reasonable to me.

[mrshu]

This has been a serious issue for me too so I'd love to see a solution too.

@glitchmr that looks very fishy!

[ridiculousfish]

With the commit 4899086 , the original problem ./a.out ^&1 | tee log should be addressed. You can assign stdout to stderr, and that happens after the pipe, so that tee sees both streams.

I want to leave this open for further discussion on syntax improvements for redirections. The current thinking is that foo | bar pipes stdout, and foo ^| bar pipes stderr. The syntax for piping both is unclear, even though there's agreement that piping both is more common than piping stderr independently.

[ridiculousfish]

My high order bit is that syntax for common redirections should be easy to learn and to remember. In particular, no common redirections should require using numbers.

Here are four proposals for piping stdout and stderr together:

1. foo ^| bar or foo |^ bar
2. foo |& bar
3. foo ^&| bar

Individual discussion:

foo ^| bar or foo |^ bar

Arguments for:

1. It's arguably intuitive. | redirects stdout, ^ redirects stderr, so ^| redirects stderr and stdout.
2. foo | bar is already a form of syntactic sugar over foo 1>| bar, which we justify by the fact that piping stdout is so common. ^ is also sugar over foo 2>| bar. So making foo ^| bar be syntactic sugar over another common operation doesn't seem so bad.
3. Piping stderr along with stdin is more common than piping stderr separately. The syntax should optimize for the most common case.

Arguments against:

1. It uses up a natural syntax for piping stderr separately. stderr can still be piped separately via foo 2>| bar.
2. It's also arguably unintuitive. You might reasonably expect that ^| would redirect stderr alone.

foo |& bar

Arguments for:

1. This is exactly the syntax used in zsh and tcsh. We could just go with that, since fish's redirections are already pretty close to POSIX syntax (and already ugly anyways). We should only break from POSIX syntax where we can improve on it.
2. It keeps the natural syntax free for redirecting stderr separately.

Arguments against:

1. It is terse but unnatural. & is not used to represent stderr anywhere else in fish.
2. It may be confused with launching a process in the background.

foo ^&| bar

1. This makes some intuitive sense. | redirects stdout, ^ redirects stderr, so ^&| means "redirect stderr and stdout."
2. It keeps the natural syntax free for redirecting stderr separately.

Arguments against:

1. It's the most verbose, and hardest to remember.

[maxfl]

If to think of '|' as only of redirection symbol with default pipe = stdout, but with possibility to override it the scheme would be the following:

1. | redirects stdout (default)

2. | redirects stdout (explicitly set)

3. ^| redirects stderr (explicitly set)
4. ^>|, >^| redirects stderr and stdout.

Arguments for:

1. For me it's intuitive.
   '>' redirects stdout by default, but 4> redirects the other output. The same for |:
   '|' redirects stdout by default, but 4>| redirects the other output.
2. It doesn't require to use additional symbols as &.

Arguments against:

1. It is unnatural for some people. In fact, naturality is a strange thing.
2. Requires additional symbol to be typed.

[anddam]

@maxfi 's proposal is probably more coherent by its usage of both symbols to represent both redirection, but I somehow don't like it and I'd rather have the ^| @ridiculousfish proposed.

If ^| were to redirect both stdout and stderr, would one redirect only stderr by using something like:

foo ^| bar >/dev/null

?

[ridiculousfish]

@anddam stderr can be piped separately via foo 2>| bar, and redirected to a file via foo ^ output.txt. The first one is gross, and the only reason it might be tolerable is if we believe that piping stderr separately is uncommon.

[zanchey]

Debian bug 520363 was concerned with the original behaviour.

[ridiculousfish]

It looks like ^| already exists, and pipes stderr along the pipeline. stdout cannot also be piped, that is, the existing behavior is that foo ^| bar | baz pipes foo.stderr to bar and bar.stdout to baz.

[najamelan]

I really don't understand this, if you try to create something less cryptic than bash, then what's wrong with:

• &stdin
• &stdout
• &stderr (or better even make them reserved keywords and ditch the &)

Can do away with the magic numbers and symbols. I would keep '>' and '>>' because that's pretty intuitive, although the difference between the two is still something that has to be learned and remembered. And I would keep '|' because it is so universally used and known, at least on *nix based systems.

cmd stderr > stdout                     # everyone to stdout
cmd stderr < stdout                     # everyone to stderr
cmd stderr > stdout > | cmd2            # everyone stdin of cmd2
cmd stderr > err.log stdout > out.log   # you get it, it needs no documentation...just need to understand english
cmd "stderr"                            # take the string stderr as an argument

[JoshCheek]

To summarize for anyone who comes here researching how to do this after me:

You can send stderr to stdout in either of these two ways:

# use the bash syntax (output is out of order b/c Ruby buffers stdout)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' 2>&1 | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
PIPED: "b\na\n"

# redirect stderr to stdout's file descriptor (in the subshell, it's connected to the pipeline)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' ^/dev/stdout | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
PIPED: "b\na\n"

And you can send stderr to the pipe instead of stdout by using ^| (note: out goes to process's out, so they aren't swapped, just hooked err to pipe instead of out)

# normal: err goes to process's err (prints to screen), out goes through pipeline
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
b
PIPED: "a\n"

# with ^|, err goes into pipelline, out goes to process's out (prints to screen)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' ^| ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
a
PIPED: "b\n"

[alphapapa]

Just a thought: Bash uses &> to redirect both STDOUT and STDERR, and |& (yes, reversed...) to pipe both STDOUT and STDERR.

So I propose that fish use &> to redirect both STDOUT and STDERR, and both |& and &| to pipe them. (I propose using both pipe forms because, hey, who can remember whether it's "and-pipe" or "pipe-and"? But remembering that they go together to "pipe-stdout-and-stderr" makes sense.)

So these would be equivalent:

Bash:

command |& grep

Fish:

command |& grep
command &| grep

[solson]

function foo
    echo stdout
    echo >&2 stderr
end

foo ^&1 | tee out # out contains both lines of output

This works for me today (version 2.2.0). Both stdout and stderr get piped to tee. I tested with a little C program as well and it still worked. Maybe this should be closed?

[faho]

Doing @gustafj's test, I now get:

>  cmd | cat - >/dev/null
stderr
> cmd 2>| cat - > /dev/null # all "2>" can also be replaced with "^"
stdout
> cmd 2>| cat -
stdout
stderr
> cmd 2>&1 | cat - > /dev/null
>

This means it's okay since probably 4899086.

However, as @ridiculousfish pointed out:

I want to leave this open for further discussion on syntax improvements for redirections. The current thinking is that foo | bar pipes stdout, and foo ^| bar pipes stderr. The syntax for piping both is unclear, even though there's agreement that piping both is more common than piping stderr independently.

Personally, I don't really care. 2>&1 | is okay for the few times I use it, though I might be weird - I pipe them separately more often than combined, including just stderr.

Because of the repurposing of this issue, I'll change the title to something more fitting.

[CamilleScholtz]

I'd like it if something like ^> and/or >^ would redirect both stdin and stderr.

[anddam]

@onodera-punpun what about the current ^> stderr redirection?

Adding a new >^ syntax would indeed be confusing.

[solson]

@anddam What? ^> is currently a syntax error.

[CamilleScholtz]

@anddam tbh I never know, I'm pretty sure it's either >&2 or ^&1, anyways, both are ugly, confusing, inconsistent, and hard to remember, ^> and >^ are both not in use currently, and the opposite of the points I listed earlier.

[anddam]

@solson correct, I mixed up the syntax (that I've been using till an hour ago) and somehow thought ^ was ^> in fact. My bad and sorry for the noise.

Anyway as a general principle I wouldn't overload too much combinations of > ^ and so, I'd rather have a more verbose but clear syntax than having to check man every time (as in fact I did with bash and what made me look for a shell with a tidier syntax).

[alphapapa]

To summarize the current suggestions:

• Both &| and |& would pipe STDOUT and STDERR together.
• Both >^ and ^> would redirect STDOUT and STDERR together.

Using both combinations for each case solves the problem of having to remember the order of the characters.

In a sense they mirror each other and seem to make sense.

However, the piping syntax is definitely a Bashism. So even though I generally favor it, let me add a third suggestion:

• Using >^| and ^>| would pipe STDOUT and STDERR together. This makes for a 3-character combo, which I guess is unusual, but it seems to make sense in that it composes existing Fish syntax characters in a logical way.

However, this also raises the question of whether something like >| or ^| should work, for consistency with the composability. Of course, >| is redundant, as plain | pipes STDOUT, but what should ^| do? I guess it would make sense for it to pipe only STDERR, as in ^&1 >/dev/null |.

And that actually seems like it could be pretty handy; it sure saves a lot of typing in that case and is much clearer as to its purpose.

It would also help avoid the issue of redirecting STDERR and STDOUT in the wrong order, which is a common gotcha, as doing >/dev/null ^&1 | does not produce the desired result, while ^&1 >/dev/null | does.

[solson]

@alphapapa >| and ^| already work. Additionally, there is an existing expanded form of N>|.

>| is equivalent to 1>| and ^| is equivalent to 2>|.

I would support having >^/^> for double-redirection to a file and >^|/^>| for double-redirection to a pipe. These seem like a natural extension of the existing syntax to me.

[alphapapa]

| and ^| already work.

Well, how about that! :D

I would support having >^/^> for double-redirection to a file and >^|/^>| for double-redirection to a pipe.

I'd just like to point out how fishy some of those look, which I hope could be preserved in the docs somehow... ;)

Or maybe we could go all-in and have >^^> redirect to /dev/null and call it the Shark Operator, because it eats the output. :D (That's actually not a bad idea...)

[balupton]

could someone add an answer on http://unix.stackexchange.com/q/70963/50703 with what the fish equivalents are for each one, cheers - as I'm having a hard time figuring out what are proposals here and what is implemented, and then for the implemented stuff, what they actually do

[ridiculousfish]

fish has the same syntax for all of those cases, except that |& and &> are not supported. I can add that to the SO question if that would be useful.

[balupton]

@ridiculousfish yeah - I was wrestling with &> myself - I ended up going with > /dev/null ^ /dev/null - the ^ /dev/null was not mentioned on the SO answer, so would be a good addition

[balupton]

with my above answer, the command > /dev/null ^ /dev/null works only for the /dev/null output - if you want to pipe stdout and stderr to the same file, it won't work, as the > and ^ will overwrite each other.

[Phidica]

Being that #4394 has removed ^ as the operator for redirection of the stderr file descriptor, I think that rules out all suggestions which involved the caret (leaving, for the most part, just things involving &). I have a new suggestion which as far as I can tell hasn't been brought up in GitHub discussions before.

How about, to redirect multiple file descriptors to the same file, we extend the syntax to allow a list of file descriptors into the redirection operator? To redirect both stdout and stderr to the same file, it would simply be 1,2>myfile. I think this feels quite intuitive, as it evokes the comma used in brace expansion, implying the behaviour will be something like {1,2}>myfile = 1>myfile 2>myfile.

Of course, this cannot actually "expand" to 1>myfile 2>myfile because as @balupton has pointed out above this (unintuitively) causes each file descriptor to independently overwrite the file. It would have to result in 1>myfile 2>&1 or 2>myfile 1>&2 (both are equivalent), but the idea is there. Understanding brace expansion lends to an understanding of "redirection expansion" and the naive theory of what it would achieve.

This applies similarly in the case of redirection-into-pipe: 1,2>| is suggestive of 1>| 2>| (actually an invalid syntax) but it will achieve 2>&1 |.

In discussion above, @ridiculousfish's take is that "no common redirections should require using numbers". But with the removal of ^, redirection of stderr explicitly requires 2, so we can't really avoid numbers in common redirections any more. In order to desire redirection of stdout and stderr, a user must already know about stderr and hence already know that it is represented by 2.

Discussion in this thread supports &> or |& as the "1 and 2" redirection or pipe. But that would effectively limit us to only having a convenient syntax to pair 1 with 2. #3303 is an open discussion of redirection involving file descriptors 3 and above, perhaps in some way where they just open on demand when first used. What if we want to pair redirection of 1 with 3? 2 with 4? I think that a list syntax (1,3>, 2,4>) would be a good generic syntax that is prepared for a possible future where redirection with higher file descriptors is more practical.

(And if we really wanted to save another character, then 1 could be implicit around a comma. So 1,2> could be done with ,2> or 2,> which are then just one character away from &>.)

[JoshCheek]

A bit late, but a note for @balupton, you can get it to work for files with

$ bash -c 'echo out; echo err >&2' > file ^ /dev/stdout`
$ cat file
out
err

[balupton]

What is the status of this now that #4394 landed in v3

[faho]

Yup, this should be closed.

An overview of the current situation:

• ^ is deprecated and can already be turned off. Use 2>.
• 2>| pipes just stderr.
• 2>&1 | pipes both. (In my experience, this is rarely necessary)

[Phidica]

Should I open a new issue with my suggestion then? I think it's an interesting idea for extending the redirection syntax

[faho]

Honestly, I don't think shortcuts are needed here. Which is also why we stepped away from ^ - it looks a teensy bit nicer, at the cost of an additional special character.

If you want to redirect both stdout and stderr, use >/path/to/file 2>&1.

[floam]

Well, I like the {1,2}> idea. It's actually consistent with expansion. It's easier to understand than the pointing 2 at &1 stuff IMO. I've actually tried exactly that and hoped it'd work in the past. I say open up an issue!

[frankseide]

I have been reading several of these issues, and I still cannot figure out how to redirect both stdout and stderr into tee in the fish shell.

[faho]

@frankseide In fish 3.1 &| is shorthand.

Otherwise 2>&1 |.

So:

whatever 2>&1 | tee
# or
whatever &| tee

---

In future, please don't comment on random issues. Instead open a new one or ask on https://gitter.im/fish-shell/fish-shell.