New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Syntax improvements for redirection & pipes #110

Open
dhasenan opened this Issue Jun 11, 2012 · 39 comments

Comments

Projects
None yet
@dhasenan

dhasenan commented Jun 11, 2012

Assuming I have a process that writes to both stderr and stdout, and I want to pipe both to tee, I should be able to do:

./a.out ^&1 | tee log

This ends up with 'log' being empty.

I suspect that fish is assigning the stderr input stream to the stdout sink, then assigning the stdout input stream to the pipe sink. Another layer of indirection would likely solve this -- instead of copying the stdout sink, you assign a wrapper sink instead. This would require caution to catch the case of redirecting >&2 ^&1, of course, but it would allow people to redirect and pipe at the same time and make it work.

@dag

This comment has been minimized.

Show comment
Hide comment
@dag

dag Jan 18, 2013

Contributor

I ran into this issue almost immediately so I'd love to see a solution.

Contributor

dag commented Jan 18, 2013

I ran into this issue almost immediately so I'd love to see a solution.

@JanKanis

This comment has been minimized.

Show comment
Hide comment
@JanKanis

JanKanis Jan 18, 2013

Member

Can anyone give or link to an exact description of how bash handles the interaction of redirections and pipes? I can't find it in the bash reference manual. I always thought the bash equivalent of the o.p.'s command would give the same result as fish, but it turns out that command 2>&1 | tee myfile sends both stdout and stder of command to tee.

My (apparently wrong) mental model of redirection in bash is that cmd1 2>&1 | cmd2 does:

  • assign the value of fd 1 (the terminal) to cmd1's fd2
  • assign the value of the pipe to cmd1's fd1.
    Fish (afaik) works like that, but what does bash do exactly?
Member

JanKanis commented Jan 18, 2013

Can anyone give or link to an exact description of how bash handles the interaction of redirections and pipes? I can't find it in the bash reference manual. I always thought the bash equivalent of the o.p.'s command would give the same result as fish, but it turns out that command 2>&1 | tee myfile sends both stdout and stder of command to tee.

My (apparently wrong) mental model of redirection in bash is that cmd1 2>&1 | cmd2 does:

  • assign the value of fd 1 (the terminal) to cmd1's fd2
  • assign the value of the pipe to cmd1's fd1.
    Fish (afaik) works like that, but what does bash do exactly?
@gustafj

This comment has been minimized.

Show comment
Hide comment
@gustafj

gustafj Jan 23, 2013

Contributor

In the manual for Bash under Pipelines i found this:
This connection is performed before any redirections specified by the command.
see http://www.gnu.org/software/bash/manual/bashref.html#Pipelines

So the redirection would be like this if my understanding is correct:

  • connect the fd1 to the pipe
  • assign the value of fd1 (the pipe) to cmd1's fd2

It is possible to do this today in fish using special syntax 2>| ex command 2>| tee myfile
In Bash this is equivalent to |& which is just syntactic sugar for 2>&1 |

I did a few trials using fish, bash and tcsh (note that my tcsh dont support to only redirect stderr, as noted in that cmd):

cmd is perl -e 'print {*STDOUT} "stdout\n"; print {*STDERR} "stderr\n";' in all below:

tcsh > cmd | cat - > /dev/null
stderr
tcsh > cmd |& cat - > /dev/null
tcsh > cmd |& cat -
stderr
stdout
tcsh > "The  shell  cannot  presently  redirect  diagnostic  output without also redirecting standard output"

bash-4.1$ cmd | cat - > /dev/null
stderr
bash-4.1$ cmd |& cat - > /dev/null
bash-4.1$ cmd |& cat -
stderr
stdout
bash-4.1$ cmd 2>&1 | cat - > /dev/null


fish > cmd | cat - > /dev/null
stderr
fish > cmd 2>| cat - > /dev/null
fish > cmd 2>| cat -
stdout
stderr
fish > cmd 2>&1 | cat - > /dev/null 
stderr
Contributor

gustafj commented Jan 23, 2013

In the manual for Bash under Pipelines i found this:
This connection is performed before any redirections specified by the command.
see http://www.gnu.org/software/bash/manual/bashref.html#Pipelines

So the redirection would be like this if my understanding is correct:

  • connect the fd1 to the pipe
  • assign the value of fd1 (the pipe) to cmd1's fd2

It is possible to do this today in fish using special syntax 2>| ex command 2>| tee myfile
In Bash this is equivalent to |& which is just syntactic sugar for 2>&1 |

I did a few trials using fish, bash and tcsh (note that my tcsh dont support to only redirect stderr, as noted in that cmd):

cmd is perl -e 'print {*STDOUT} "stdout\n"; print {*STDERR} "stderr\n";' in all below:

tcsh > cmd | cat - > /dev/null
stderr
tcsh > cmd |& cat - > /dev/null
tcsh > cmd |& cat -
stderr
stdout
tcsh > "The  shell  cannot  presently  redirect  diagnostic  output without also redirecting standard output"

bash-4.1$ cmd | cat - > /dev/null
stderr
bash-4.1$ cmd |& cat - > /dev/null
bash-4.1$ cmd |& cat -
stderr
stdout
bash-4.1$ cmd 2>&1 | cat - > /dev/null


fish > cmd | cat - > /dev/null
stderr
fish > cmd 2>| cat - > /dev/null
fish > cmd 2>| cat -
stdout
stderr
fish > cmd 2>&1 | cat - > /dev/null 
stderr
@dag

This comment has been minimized.

Show comment
Hide comment
@dag

dag Jan 23, 2013

Contributor

My expectation would be that 2>| would pipe stderr but not stdout. And also that if that syntax works then so should ^|, doing the same. Piping both is likely more useful, but might arguably break "The law of user focus" in that it's not intuitive.

Contributor

dag commented Jan 23, 2013

My expectation would be that 2>| would pipe stderr but not stdout. And also that if that syntax works then so should ^|, doing the same. Piping both is likely more useful, but might arguably break "The law of user focus" in that it's not intuitive.

@gustafj

This comment has been minimized.

Show comment
Hide comment
@gustafj

gustafj Jan 23, 2013

Contributor

I like the ^| syntax, feels more "fish-like", and i also agree that when using 2>| (or ^|) one could expect it to only pipe stderr.
But then what syntax would be appropriate to use when piping both (which is probably what you most often want, except for when only piping stdout).

Proposal:

  • Only stdout |, >| or 1>|
  • Only stderr ^| or 2>|
  • Both stderr & stdout ^>| or >^| or any combination of redirection followed by pipe, ex: ^&1 | or >&2 ^|
Contributor

gustafj commented Jan 23, 2013

I like the ^| syntax, feels more "fish-like", and i also agree that when using 2>| (or ^|) one could expect it to only pipe stderr.
But then what syntax would be appropriate to use when piping both (which is probably what you most often want, except for when only piping stdout).

Proposal:

  • Only stdout |, >| or 1>|
  • Only stderr ^| or 2>|
  • Both stderr & stdout ^>| or >^| or any combination of redirection followed by pipe, ex: ^&1 | or >&2 ^|
@dag

This comment has been minimized.

Show comment
Hide comment
@dag

dag Jan 23, 2013

Contributor

The good thing is that the syntax is beginning to look like a fish ...

Contributor

dag commented Jan 23, 2013

The good thing is that the syntax is beginning to look like a fish ...

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Jun 28, 2013

Member

Let’s try to fix this one in next-minor

Member

ridiculousfish commented Jun 28, 2013

Let’s try to fix this one in next-minor

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Aug 18, 2013

Member

If you can pipe stdout and stderr independently, how do pipes associate? For example:

foo ^| bar | baz

Does baz get the output of foo or bar? Anyone know how other shells handle this?

Member

ridiculousfish commented Aug 18, 2013

If you can pipe stdout and stderr independently, how do pipes associate? For example:

foo ^| bar | baz

Does baz get the output of foo or bar? Anyone know how other shells handle this?

@timbertson

This comment has been minimized.

Show comment
Hide comment
@timbertson

timbertson Aug 18, 2013

Contributor

If we had subshells, I assume the answer would be that you'd need to do the following if you want stderr to go through multiple processes:

$ foo ^| (bar | baz) | frob

stderr goes through baz | bar, while stdout goes through frob. I thought the above would be possible in bash, but it seems you only get one pipe in bash (http://stackoverflow.com/questions/2342826/how-to-pipe-stderr-and-not-stdout)

Is there any subshell-like thing in fish? I often want to do stuff in another dir temporarily, which I used to do in bash with (cd DIR; do-something). I miss that.

Contributor

timbertson commented Aug 18, 2013

If we had subshells, I assume the answer would be that you'd need to do the following if you want stderr to go through multiple processes:

$ foo ^| (bar | baz) | frob

stderr goes through baz | bar, while stdout goes through frob. I thought the above would be possible in bash, but it seems you only get one pipe in bash (http://stackoverflow.com/questions/2342826/how-to-pipe-stderr-and-not-stdout)

Is there any subshell-like thing in fish? I often want to do stuff in another dir temporarily, which I used to do in bash with (cd DIR; do-something). I miss that.

@xfix

This comment has been minimized.

Show comment
Hide comment
@xfix

xfix Aug 18, 2013

Member

@gfxmonk: But we do.

foo ^| begin
    bar | baz
end | frob
Member

xfix commented Aug 18, 2013

@gfxmonk: But we do.

foo ^| begin
    bar | baz
end | frob
@anddam

This comment has been minimized.

Show comment
Hide comment
@anddam

anddam Aug 18, 2013

@glitchmr that's brilliant

anddam commented Aug 18, 2013

@glitchmr that's brilliant

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Aug 18, 2013

Member

It sounds like the proposed answer to my question is that a stderr-only pipe would be standalone, and not part of the larger pipeline. That is, foo ^| bar | baz is foo ^| bar and foo | baz. If you wanted to pipe stderr further, you use begin/end to create a block. That seems reasonable to me.

Member

ridiculousfish commented Aug 18, 2013

It sounds like the proposed answer to my question is that a stderr-only pipe would be standalone, and not part of the larger pipeline. That is, foo ^| bar | baz is foo ^| bar and foo | baz. If you wanted to pipe stderr further, you use begin/end to create a block. That seems reasonable to me.

@mrshu

This comment has been minimized.

Show comment
Hide comment
@mrshu

mrshu Aug 19, 2013

Contributor

This has been a serious issue for me too so I'd love to see a solution too.

@glitchmr that looks very fishy!

Contributor

mrshu commented Aug 19, 2013

This has been a serious issue for me too so I'd love to see a solution too.

@glitchmr that looks very fishy!

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Aug 20, 2013

Member

With the commit 4899086 , the original problem ./a.out ^&1 | tee log should be addressed. You can assign stdout to stderr, and that happens after the pipe, so that tee sees both streams.

I want to leave this open for further discussion on syntax improvements for redirections. The current thinking is that foo | bar pipes stdout, and foo ^| bar pipes stderr. The syntax for piping both is unclear, even though there's agreement that piping both is more common than piping stderr independently.

Member

ridiculousfish commented Aug 20, 2013

With the commit 4899086 , the original problem ./a.out ^&1 | tee log should be addressed. You can assign stdout to stderr, and that happens after the pipe, so that tee sees both streams.

I want to leave this open for further discussion on syntax improvements for redirections. The current thinking is that foo | bar pipes stdout, and foo ^| bar pipes stderr. The syntax for piping both is unclear, even though there's agreement that piping both is more common than piping stderr independently.

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Aug 20, 2013

Member

My high order bit is that syntax for common redirections should be easy to learn and to remember. In particular, no common redirections should require using numbers.

Here are four proposals for piping stdout and stderr together:

  1. foo ^| bar or foo |^ bar
  2. foo |& bar
  3. foo ^&| bar

Individual discussion:

foo ^| bar or foo |^ bar

Arguments for:

  1. It's arguably intuitive. | redirects stdout, ^ redirects stderr, so ^| redirects stderr and stdout.
  2. foo | bar is already a form of syntactic sugar over foo 1>| bar, which we justify by the fact that piping stdout is so common. ^ is also sugar over foo 2>| bar. So making foo ^| bar be syntactic sugar over another common operation doesn't seem so bad.
  3. Piping stderr along with stdin is more common than piping stderr separately. The syntax should optimize for the most common case.

Arguments against:

  1. It uses up a natural syntax for piping stderr separately. stderr can still be piped separately via foo 2>| bar.
  2. It's also arguably unintuitive. You might reasonably expect that ^| would redirect stderr alone.
foo |& bar

Arguments for:

  1. This is exactly the syntax used in zsh and tcsh. We could just go with that, since fish's redirections are already pretty close to POSIX syntax (and already ugly anyways). We should only break from POSIX syntax where we can improve on it.
  2. It keeps the natural syntax free for redirecting stderr separately.

Arguments against:

  1. It is terse but unnatural. & is not used to represent stderr anywhere else in fish.
  2. It may be confused with launching a process in the background.
foo ^&| bar
  1. This makes some intuitive sense. | redirects stdout, ^ redirects stderr, so ^&| means "redirect stderr and stdout."
  2. It keeps the natural syntax free for redirecting stderr separately.

Arguments against:

  1. It's the most verbose, and hardest to remember.
Member

ridiculousfish commented Aug 20, 2013

My high order bit is that syntax for common redirections should be easy to learn and to remember. In particular, no common redirections should require using numbers.

Here are four proposals for piping stdout and stderr together:

  1. foo ^| bar or foo |^ bar
  2. foo |& bar
  3. foo ^&| bar

Individual discussion:

foo ^| bar or foo |^ bar

Arguments for:

  1. It's arguably intuitive. | redirects stdout, ^ redirects stderr, so ^| redirects stderr and stdout.
  2. foo | bar is already a form of syntactic sugar over foo 1>| bar, which we justify by the fact that piping stdout is so common. ^ is also sugar over foo 2>| bar. So making foo ^| bar be syntactic sugar over another common operation doesn't seem so bad.
  3. Piping stderr along with stdin is more common than piping stderr separately. The syntax should optimize for the most common case.

Arguments against:

  1. It uses up a natural syntax for piping stderr separately. stderr can still be piped separately via foo 2>| bar.
  2. It's also arguably unintuitive. You might reasonably expect that ^| would redirect stderr alone.
foo |& bar

Arguments for:

  1. This is exactly the syntax used in zsh and tcsh. We could just go with that, since fish's redirections are already pretty close to POSIX syntax (and already ugly anyways). We should only break from POSIX syntax where we can improve on it.
  2. It keeps the natural syntax free for redirecting stderr separately.

Arguments against:

  1. It is terse but unnatural. & is not used to represent stderr anywhere else in fish.
  2. It may be confused with launching a process in the background.
foo ^&| bar
  1. This makes some intuitive sense. | redirects stdout, ^ redirects stderr, so ^&| means "redirect stderr and stdout."
  2. It keeps the natural syntax free for redirecting stderr separately.

Arguments against:

  1. It's the most verbose, and hardest to remember.
@maxfl

This comment has been minimized.

Show comment
Hide comment
@maxfl

maxfl Aug 21, 2013

Contributor

If to think of '|' as only of redirection symbol with default pipe = stdout, but with possibility to override it the scheme would be the following:

  1. | redirects stdout (default)
  2. | redirects stdout (explicitly set)

  3. ^| redirects stderr (explicitly set)
  4. ^>|, >^| redirects stderr and stdout.

Arguments for:

  1. For me it's intuitive.
    '>' redirects stdout by default, but 4> redirects the other output. The same for |:
    '|' redirects stdout by default, but 4>| redirects the other output.
  2. It doesn't require to use additional symbols as &.

Arguments against:

  1. It is unnatural for some people. In fact, naturality is a strange thing.
  2. Requires additional symbol to be typed.
Contributor

maxfl commented Aug 21, 2013

If to think of '|' as only of redirection symbol with default pipe = stdout, but with possibility to override it the scheme would be the following:

  1. | redirects stdout (default)
  2. | redirects stdout (explicitly set)

  3. ^| redirects stderr (explicitly set)
  4. ^>|, >^| redirects stderr and stdout.

Arguments for:

  1. For me it's intuitive.
    '>' redirects stdout by default, but 4> redirects the other output. The same for |:
    '|' redirects stdout by default, but 4>| redirects the other output.
  2. It doesn't require to use additional symbols as &.

Arguments against:

  1. It is unnatural for some people. In fact, naturality is a strange thing.
  2. Requires additional symbol to be typed.
@anddam

This comment has been minimized.

Show comment
Hide comment
@anddam

anddam Aug 21, 2013

@maxfi 's proposal is probably more coherent by its usage of both symbols to represent both redirection, but I somehow don't like it and I'd rather have the ^| @ridiculousfish proposed.

If ^| were to redirect both stdout and stderr, would one redirect only stderr by using something like:

foo ^| bar >/dev/null

?

anddam commented Aug 21, 2013

@maxfi 's proposal is probably more coherent by its usage of both symbols to represent both redirection, but I somehow don't like it and I'd rather have the ^| @ridiculousfish proposed.

If ^| were to redirect both stdout and stderr, would one redirect only stderr by using something like:

foo ^| bar >/dev/null

?

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Aug 21, 2013

Member

@anddam stderr can be piped separately via foo 2>| bar, and redirected to a file via foo ^ output.txt. The first one is gross, and the only reason it might be tolerable is if we believe that piping stderr separately is uncommon.

Member

ridiculousfish commented Aug 21, 2013

@anddam stderr can be piped separately via foo 2>| bar, and redirected to a file via foo ^ output.txt. The first one is gross, and the only reason it might be tolerable is if we believe that piping stderr separately is uncommon.

@zanchey

This comment has been minimized.

Show comment
Hide comment
@zanchey

zanchey Sep 1, 2013

Member

Debian bug 520363 was concerned with the original behaviour.

Member

zanchey commented Sep 1, 2013

Debian bug 520363 was concerned with the original behaviour.

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Oct 13, 2013

Member

It looks like ^| already exists, and pipes stderr along the pipeline. stdout cannot also be piped, that is, the existing behavior is that foo ^| bar | baz pipes foo.stderr to bar and bar.stdout to baz.

Member

ridiculousfish commented Oct 13, 2013

It looks like ^| already exists, and pipes stderr along the pipeline. stdout cannot also be piped, that is, the existing behavior is that foo ^| bar | baz pipes foo.stderr to bar and bar.stdout to baz.

haarts pushed a commit to haarts/fish-shell that referenced this issue Nov 1, 2013

Big fat refactoring of how redirections work. In fish 1.x and 2.0.0, …
…the redirections for a process were flattened into a big list associated with the job, so there was no way to tell which redirections applied to each process. Each process therefore got all the redirections associated with the job. See fish-shell#877 for how this could manifest.

With this change, jobs only track their block-level redirections. Process level redirections are correctly associated with the process, and at exec time we stitch them together (block, pipe, and process redirects).

This fixes the weird issues where redirects bleed across pipelines (like #877), and also allows us to play with the order in which redirections are applied, since the final list is constructed right before it's needed.  This lets us put pipes after block level redirections but before process level redirections, so that a 2>&1-type redirection gets picked up after the pipe, i.e. it should fix fish-shell#110

This is a significant change. The tests all pass. Cross your fingers.
@najamelan

This comment has been minimized.

Show comment
Hide comment
@najamelan

najamelan Apr 21, 2014

I really don't understand this, if you try to create something less cryptic than bash, then what's wrong with:

  • &stdin
  • &stdout
  • &stderr (or better even make them reserved keywords and ditch the &)

Can do away with the magic numbers and symbols. I would keep '>' and '>>' because that's pretty intuitive, although the difference between the two is still something that has to be learned and remembered. And I would keep '|' because it is so universally used and known, at least on *nix based systems.

cmd stderr > stdout                     # everyone to stdout
cmd stderr < stdout                     # everyone to stderr
cmd stderr > stdout > | cmd2            # everyone stdin of cmd2
cmd stderr > err.log stdout > out.log   # you get it, it needs no documentation...just need to understand english
cmd "stderr"                            # take the string stderr as an argument

najamelan commented Apr 21, 2014

I really don't understand this, if you try to create something less cryptic than bash, then what's wrong with:

  • &stdin
  • &stdout
  • &stderr (or better even make them reserved keywords and ditch the &)

Can do away with the magic numbers and symbols. I would keep '>' and '>>' because that's pretty intuitive, although the difference between the two is still something that has to be learned and remembered. And I would keep '|' because it is so universally used and known, at least on *nix based systems.

cmd stderr > stdout                     # everyone to stdout
cmd stderr < stdout                     # everyone to stderr
cmd stderr > stdout > | cmd2            # everyone stdin of cmd2
cmd stderr > err.log stdout > out.log   # you get it, it needs no documentation...just need to understand english
cmd "stderr"                            # take the string stderr as an argument
@JoshCheek

This comment has been minimized.

Show comment
Hide comment
@JoshCheek

JoshCheek Dec 31, 2014

To summarize for anyone who comes here researching how to do this after me:

You can send stderr to stdout in either of these two ways:

# use the bash syntax (output is out of order b/c Ruby buffers stdout)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' 2>&1 | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
PIPED: "b\na\n"

# redirect stderr to stdout's file descriptor (in the subshell, it's connected to the pipeline)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' ^/dev/stdout | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
PIPED: "b\na\n"

And you can send stderr to the pipe instead of stdout by using ^| (note: out goes to process's out, so they aren't swapped, just hooked err to pipe instead of out)

# normal: err goes to process's err (prints to screen), out goes through pipeline
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
b
PIPED: "a\n"

# with ^|, err goes into pipelline, out goes to process's out (prints to screen)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' ^| ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
a
PIPED: "b\n"

JoshCheek commented Dec 31, 2014

To summarize for anyone who comes here researching how to do this after me:

You can send stderr to stdout in either of these two ways:

# use the bash syntax (output is out of order b/c Ruby buffers stdout)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' 2>&1 | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
PIPED: "b\na\n"

# redirect stderr to stdout's file descriptor (in the subshell, it's connected to the pipeline)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' ^/dev/stdout | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
PIPED: "b\na\n"

And you can send stderr to the pipe instead of stdout by using ^| (note: out goes to process's out, so they aren't swapped, just hooked err to pipe instead of out)

# normal: err goes to process's err (prints to screen), out goes through pipeline
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' | ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
b
PIPED: "a\n"

# with ^|, err goes into pipelline, out goes to process's out (prints to screen)
> ruby -e '$stdout.puts "a"; $stderr.puts "b"' ^| ruby -e 'puts "PIPED: #{$stdin.read.inspect}"'
a
PIPED: "b\n"
@alphapapa

This comment has been minimized.

Show comment
Hide comment
@alphapapa

alphapapa Apr 2, 2015

Just a thought: Bash uses &> to redirect both STDOUT and STDERR, and |& (yes, reversed...) to pipe both STDOUT and STDERR.

So I propose that fish use &> to redirect both STDOUT and STDERR, and both |& and &| to pipe them. (I propose using both pipe forms because, hey, who can remember whether it's "and-pipe" or "pipe-and"? But remembering that they go together to "pipe-stdout-and-stderr" makes sense.)

So these would be equivalent:

Bash:

command |& grep

Fish:

command |& grep
command &| grep

alphapapa commented Apr 2, 2015

Just a thought: Bash uses &> to redirect both STDOUT and STDERR, and |& (yes, reversed...) to pipe both STDOUT and STDERR.

So I propose that fish use &> to redirect both STDOUT and STDERR, and both |& and &| to pipe them. (I propose using both pipe forms because, hey, who can remember whether it's "and-pipe" or "pipe-and"? But remembering that they go together to "pipe-stdout-and-stderr" makes sense.)

So these would be equivalent:

Bash:

command |& grep

Fish:

command |& grep
command &| grep
@solson

This comment has been minimized.

Show comment
Hide comment
@solson

solson Nov 6, 2015

function foo
    echo stdout
    echo >&2 stderr
end

foo ^&1 | tee out # out contains both lines of output

This works for me today (version 2.2.0). Both stdout and stderr get piped to tee. I tested with a little C program as well and it still worked. Maybe this should be closed?

solson commented Nov 6, 2015

function foo
    echo stdout
    echo >&2 stderr
end

foo ^&1 | tee out # out contains both lines of output

This works for me today (version 2.2.0). Both stdout and stderr get piped to tee. I tested with a little C program as well and it still worked. Maybe this should be closed?

@faho

This comment has been minimized.

Show comment
Hide comment
@faho

faho Nov 6, 2015

Member

Doing @gustafj's test, I now get:

>  cmd | cat - >/dev/null
stderr
> cmd 2>| cat - > /dev/null # all "2>" can also be replaced with "^"
stdout
> cmd 2>| cat -
stdout
stderr
> cmd 2>&1 | cat - > /dev/null
>

This means it's okay since probably 4899086.

However, as @ridiculousfish pointed out:

I want to leave this open for further discussion on syntax improvements for redirections. The current thinking is that foo | bar pipes stdout, and foo ^| bar pipes stderr. The syntax for piping both is unclear, even though there's agreement that piping both is more common than piping stderr independently.

Personally, I don't really care. 2>&1 | is okay for the few times I use it, though I might be weird - I pipe them separately more often than combined, including just stderr.

Because of the repurposing of this issue, I'll change the title to something more fitting.

Member

faho commented Nov 6, 2015

Doing @gustafj's test, I now get:

>  cmd | cat - >/dev/null
stderr
> cmd 2>| cat - > /dev/null # all "2>" can also be replaced with "^"
stdout
> cmd 2>| cat -
stdout
stderr
> cmd 2>&1 | cat - > /dev/null
>

This means it's okay since probably 4899086.

However, as @ridiculousfish pointed out:

I want to leave this open for further discussion on syntax improvements for redirections. The current thinking is that foo | bar pipes stdout, and foo ^| bar pipes stderr. The syntax for piping both is unclear, even though there's agreement that piping both is more common than piping stderr independently.

Personally, I don't really care. 2>&1 | is okay for the few times I use it, though I might be weird - I pipe them separately more often than combined, including just stderr.

Because of the repurposing of this issue, I'll change the title to something more fitting.

@faho faho changed the title from Redirect stderr to stdout and pipe doesn't work as expected to Syntax improvements for redirection & pipes Nov 6, 2015

@onodera-punpun

This comment has been minimized.

Show comment
Hide comment
@onodera-punpun

onodera-punpun Jul 13, 2016

Contributor

I'd like it if something like ^> and/or >^ would redirect both stdin and stderr.

Contributor

onodera-punpun commented Jul 13, 2016

I'd like it if something like ^> and/or >^ would redirect both stdin and stderr.

@anddam

This comment has been minimized.

Show comment
Hide comment
@anddam

anddam Jul 17, 2016

@onodera-punpun what about the current ^> stderr redirection?

Adding a new >^ syntax would indeed be confusing.

anddam commented Jul 17, 2016

@onodera-punpun what about the current ^> stderr redirection?

Adding a new >^ syntax would indeed be confusing.

@solson

This comment has been minimized.

Show comment
Hide comment
@solson

solson Jul 17, 2016

@anddam What? ^> is currently a syntax error.

solson commented Jul 17, 2016

@anddam What? ^> is currently a syntax error.

@onodera-punpun

This comment has been minimized.

Show comment
Hide comment
@onodera-punpun

onodera-punpun Jul 17, 2016

Contributor

@anddam tbh I never know, I'm pretty sure it's either >&2 or ^&1, anyways, both are ugly, confusing, inconsistent, and hard to remember, ^> and >^ are both not in use currently, and the opposite of the points I listed earlier.

Contributor

onodera-punpun commented Jul 17, 2016

@anddam tbh I never know, I'm pretty sure it's either >&2 or ^&1, anyways, both are ugly, confusing, inconsistent, and hard to remember, ^> and >^ are both not in use currently, and the opposite of the points I listed earlier.

@anddam

This comment has been minimized.

Show comment
Hide comment
@anddam

anddam Jul 17, 2016

@solson correct, I mixed up the syntax (that I've been using till an hour ago) and somehow thought ^ was ^> in fact. My bad and sorry for the noise.

Anyway as a general principle I wouldn't overload too much combinations of > ^ and so, I'd rather have a more verbose but clear syntax than having to check man every time (as in fact I did with bash and what made me look for a shell with a tidier syntax).

anddam commented Jul 17, 2016

@solson correct, I mixed up the syntax (that I've been using till an hour ago) and somehow thought ^ was ^> in fact. My bad and sorry for the noise.

Anyway as a general principle I wouldn't overload too much combinations of > ^ and so, I'd rather have a more verbose but clear syntax than having to check man every time (as in fact I did with bash and what made me look for a shell with a tidier syntax).

@alphapapa

This comment has been minimized.

Show comment
Hide comment
@alphapapa

alphapapa Sep 23, 2016

To summarize the current suggestions:

  • Both &| and |& would pipe STDOUT and STDERR together.
  • Both >^ and ^> would redirect STDOUT and STDERR together.

Using both combinations for each case solves the problem of having to remember the order of the characters.

In a sense they mirror each other and seem to make sense.

However, the piping syntax is definitely a Bashism. So even though I generally favor it, let me add a third suggestion:

  • Using >^| and ^>| would pipe STDOUT and STDERR together. This makes for a 3-character combo, which I guess is unusual, but it seems to make sense in that it composes existing Fish syntax characters in a logical way.

However, this also raises the question of whether something like >| or ^| should work, for consistency with the composability. Of course, >| is redundant, as plain | pipes STDOUT, but what should ^| do? I guess it would make sense for it to pipe only STDERR, as in ^&1 >/dev/null |.

And that actually seems like it could be pretty handy; it sure saves a lot of typing in that case and is much clearer as to its purpose.

It would also help avoid the issue of redirecting STDERR and STDOUT in the wrong order, which is a common gotcha, as doing >/dev/null ^&1 | does not produce the desired result, while ^&1 >/dev/null | does.

alphapapa commented Sep 23, 2016

To summarize the current suggestions:

  • Both &| and |& would pipe STDOUT and STDERR together.
  • Both >^ and ^> would redirect STDOUT and STDERR together.

Using both combinations for each case solves the problem of having to remember the order of the characters.

In a sense they mirror each other and seem to make sense.

However, the piping syntax is definitely a Bashism. So even though I generally favor it, let me add a third suggestion:

  • Using >^| and ^>| would pipe STDOUT and STDERR together. This makes for a 3-character combo, which I guess is unusual, but it seems to make sense in that it composes existing Fish syntax characters in a logical way.

However, this also raises the question of whether something like >| or ^| should work, for consistency with the composability. Of course, >| is redundant, as plain | pipes STDOUT, but what should ^| do? I guess it would make sense for it to pipe only STDERR, as in ^&1 >/dev/null |.

And that actually seems like it could be pretty handy; it sure saves a lot of typing in that case and is much clearer as to its purpose.

It would also help avoid the issue of redirecting STDERR and STDOUT in the wrong order, which is a common gotcha, as doing >/dev/null ^&1 | does not produce the desired result, while ^&1 >/dev/null | does.

@solson

This comment has been minimized.

Show comment
Hide comment
@solson

solson Sep 23, 2016

@alphapapa >| and ^| already work. Additionally, there is an existing expanded form of N>|.

>| is equivalent to 1>| and ^| is equivalent to 2>|.

I would support having >^/^> for double-redirection to a file and >^|/^>| for double-redirection to a pipe. These seem like a natural extension of the existing syntax to me.

solson commented Sep 23, 2016

@alphapapa >| and ^| already work. Additionally, there is an existing expanded form of N>|.

>| is equivalent to 1>| and ^| is equivalent to 2>|.

I would support having >^/^> for double-redirection to a file and >^|/^>| for double-redirection to a pipe. These seem like a natural extension of the existing syntax to me.

@alphapapa

This comment has been minimized.

Show comment
Hide comment
@alphapapa

alphapapa Sep 23, 2016

| and ^| already work.

Well, how about that! :D

I would support having >^/^> for double-redirection to a file and >^|/^>| for double-redirection to a pipe.

I'd just like to point out how fishy some of those look, which I hope could be preserved in the docs somehow... ;)

Or maybe we could go all-in and have >^^> redirect to /dev/null and call it the Shark Operator, because it eats the output. :D (That's actually not a bad idea...)

alphapapa commented Sep 23, 2016

| and ^| already work.

Well, how about that! :D

I would support having >^/^> for double-redirection to a file and >^|/^>| for double-redirection to a pipe.

I'd just like to point out how fishy some of those look, which I hope could be preserved in the docs somehow... ;)

Or maybe we could go all-in and have >^^> redirect to /dev/null and call it the Shark Operator, because it eats the output. :D (That's actually not a bad idea...)

@floam floam added the enhancement label Sep 23, 2016

@balupton

This comment has been minimized.

Show comment
Hide comment
@balupton

balupton Feb 24, 2017

could someone add an answer on http://unix.stackexchange.com/q/70963/50703 with what the fish equivalents are for each one, cheers - as I'm having a hard time figuring out what are proposals here and what is implemented, and then for the implemented stuff, what they actually do

balupton commented Feb 24, 2017

could someone add an answer on http://unix.stackexchange.com/q/70963/50703 with what the fish equivalents are for each one, cheers - as I'm having a hard time figuring out what are proposals here and what is implemented, and then for the implemented stuff, what they actually do

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Feb 24, 2017

Member

fish has the same syntax for all of those cases, except that |& and &> are not supported. I can add that to the SO question if that would be useful.

Member

ridiculousfish commented Feb 24, 2017

fish has the same syntax for all of those cases, except that |& and &> are not supported. I can add that to the SO question if that would be useful.

@balupton

This comment has been minimized.

Show comment
Hide comment
@balupton

balupton Feb 24, 2017

@ridiculousfish yeah - I was wrestling with &> myself - I ended up going with > /dev/null ^ /dev/null - the ^ /dev/null was not mentioned on the SO answer, so would be a good addition

balupton commented Feb 24, 2017

@ridiculousfish yeah - I was wrestling with &> myself - I ended up going with > /dev/null ^ /dev/null - the ^ /dev/null was not mentioned on the SO answer, so would be a good addition

@balupton

This comment has been minimized.

Show comment
Hide comment
@balupton

balupton Jul 3, 2017

with my above answer, the command > /dev/null ^ /dev/null works only for the /dev/null output - if you want to pipe stdout and stderr to the same file, it won't work, as the > and ^ will overwrite each other.

balupton commented Jul 3, 2017

with my above answer, the command > /dev/null ^ /dev/null works only for the /dev/null output - if you want to pipe stdout and stderr to the same file, it won't work, as the > and ^ will overwrite each other.

@Phidica

This comment has been minimized.

Show comment
Hide comment
@Phidica

Phidica Jun 20, 2018

Contributor

Being that #4394 has removed ^ as the operator for redirection of the stderr file descriptor, I think that rules out all suggestions which involved the caret (leaving, for the most part, just things involving &). I have a new suggestion which as far as I can tell hasn't been brought up in GitHub discussions before.

How about, to redirect multiple file descriptors to the same file, we extend the syntax to allow a list of file descriptors into the redirection operator? To redirect both stdout and stderr to the same file, it would simply be 1,2>myfile. I think this feels quite intuitive, as it evokes the comma used in brace expansion, implying the behaviour will be something like {1,2}>myfile = 1>myfile 2>myfile.

Of course, this cannot actually "expand" to 1>myfile 2>myfile because as @balupton has pointed out above this (unintuitively) causes each file descriptor to independently overwrite the file. It would have to result in 1>myfile 2>&1 or 2>myfile 1>&2 (both are equivalent), but the idea is there. Understanding brace expansion lends to an understanding of "redirection expansion" and the naive theory of what it would achieve.

This applies similarly in the case of redirection-into-pipe: 1,2>| is suggestive of 1>| 2>| (actually an invalid syntax) but it will achieve 2>&1 |.

In discussion above, @ridiculousfish's take is that "no common redirections should require using numbers". But with the removal of ^, redirection of stderr explicitly requires 2, so we can't really avoid numbers in common redirections any more. In order to desire redirection of stdout and stderr, a user must already know about stderr and hence already know that it is represented by 2.

Discussion in this thread supports &> or |& as the "1 and 2" redirection or pipe. But that would effectively limit us to only having a convenient syntax to pair 1 with 2. #3303 is an open discussion of redirection involving file descriptors 3 and above, perhaps in some way where they just open on demand when first used. What if we want to pair redirection of 1 with 3? 2 with 4? I think that a list syntax (1,3>, 2,4>) would be a good generic syntax that is prepared for a possible future where redirection with higher file descriptors is more practical.

(And if we really wanted to save another character, then 1 could be implicit around a comma. So 1,2> could be done with ,2> or 2,> which are then just one character away from &>.)

Contributor

Phidica commented Jun 20, 2018

Being that #4394 has removed ^ as the operator for redirection of the stderr file descriptor, I think that rules out all suggestions which involved the caret (leaving, for the most part, just things involving &). I have a new suggestion which as far as I can tell hasn't been brought up in GitHub discussions before.

How about, to redirect multiple file descriptors to the same file, we extend the syntax to allow a list of file descriptors into the redirection operator? To redirect both stdout and stderr to the same file, it would simply be 1,2>myfile. I think this feels quite intuitive, as it evokes the comma used in brace expansion, implying the behaviour will be something like {1,2}>myfile = 1>myfile 2>myfile.

Of course, this cannot actually "expand" to 1>myfile 2>myfile because as @balupton has pointed out above this (unintuitively) causes each file descriptor to independently overwrite the file. It would have to result in 1>myfile 2>&1 or 2>myfile 1>&2 (both are equivalent), but the idea is there. Understanding brace expansion lends to an understanding of "redirection expansion" and the naive theory of what it would achieve.

This applies similarly in the case of redirection-into-pipe: 1,2>| is suggestive of 1>| 2>| (actually an invalid syntax) but it will achieve 2>&1 |.

In discussion above, @ridiculousfish's take is that "no common redirections should require using numbers". But with the removal of ^, redirection of stderr explicitly requires 2, so we can't really avoid numbers in common redirections any more. In order to desire redirection of stdout and stderr, a user must already know about stderr and hence already know that it is represented by 2.

Discussion in this thread supports &> or |& as the "1 and 2" redirection or pipe. But that would effectively limit us to only having a convenient syntax to pair 1 with 2. #3303 is an open discussion of redirection involving file descriptors 3 and above, perhaps in some way where they just open on demand when first used. What if we want to pair redirection of 1 with 3? 2 with 4? I think that a list syntax (1,3>, 2,4>) would be a good generic syntax that is prepared for a possible future where redirection with higher file descriptors is more practical.

(And if we really wanted to save another character, then 1 could be implicit around a comma. So 1,2> could be done with ,2> or 2,> which are then just one character away from &>.)

@JoshCheek

This comment has been minimized.

Show comment
Hide comment
@JoshCheek

JoshCheek Jun 20, 2018

A bit late, but a note for @balupton, you can get it to work for files with

$ bash -c 'echo out; echo err >&2' > file ^ /dev/stdout`
$ cat file
out
err

JoshCheek commented Jun 20, 2018

A bit late, but a note for @balupton, you can get it to work for files with

$ bash -c 'echo out; echo err >&2' > file ^ /dev/stdout`
$ cat file
out
err
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment