New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipe stdout and stderr to two separate commands #485

Closed
mqudsi opened this Issue Sep 19, 2017 · 7 comments

Comments

Projects
None yet
3 participants
@mqudsi

mqudsi commented Sep 19, 2017

I haven't dug into this too deeply, but by looking through the documentation and testing a few things, it does not seem that it is possible to operate on both stdout and stderr streams simultaneously (but separately)

~>sh -c "echo stdout; echo 1>&2 stderr"
stdout
stderr

~>sh -c "echo stdout; echo 1>&2 stderr" | (or ?(egrep nostdout) true)
stderr

~> sh -c "echo stdout; echo 1>&2 stderr" | or ?(egrep nostdout) ?()
stderr
▶ $ok

~/rand> sh -c "echo stdout; echo 1>&2 stderr" | or ?(egrep nostdout) () 2>&1 | or ?(egrep nostderr) ()
stderr
▶ ?(fail 'egrep exited with 1')

Is it not possible to redirect both?

@xiaq

This comment has been minimized.

Member

xiaq commented Sep 30, 2017

The pipe symbol always connect stdout of the previous command with the stdin of the next command. stderr does not participate in pipelining; I do not know of any shells where stderr does.

@mqudsi

This comment has been minimized.

mqudsi commented Oct 1, 2017

@xiaq thanks, I supposed as much. Powershell can do it if you use the process api, but that's mainly cheating.

I cannot stress how amazing (and unique) it would be if it were possible to come up with a nice, elegant way of operating on both stderr and stdout separately, without blocking, in elvish. It would be the killer feature everyone needs and they don't even realize it, and elvish would be the only shell to provide that out-of-the-box.

The primary issue with with working on both is parsing intent. Presuming stream redirection operators like 1>| and 2>|, how the command

./foo 1>| bar 2>| bar2

is parsed is ambiguous. Is the stderror redirection meant to apply to ./foo or the output of bar? This can only be solved by using brackets:

./foo 1>| bar 2>| bar2
./foo 1>| { bar 2>| bar2 }
@xiaq

This comment has been minimized.

Member

xiaq commented Oct 11, 2017

@mqudsi Thanks for clarifying things. For new shell features, it is often much more tricky to come up with a sensible syntax than a sensible semantics. Welcome to shell design :)

There is a more conservative approach though. Elvish already supports building and manipulating pipes with the pipe, prclose and pwclose commands. Redirecting the input to a pipe will use the read end, while redirecting the output will use the write end.

What Elvish still lacks is a construct for running several things in parallel, and wait for them to finish. The pipeline is actually very close to such a construct: in a | b | c, a, b and c are run in parallel, and the pipeline waits for all of them to finish. However, the pipeline also connects the I/O in a standard manner (stdout of a to stdin of b, and stdout of b to stdin of c), and in the use case under question we want to override this behaviour.

What can work is a (say) builtin run-parallel that takes lambdas, runs them in parallel and wait for them to finish. For instance, run-parallel { a } { b } { c } does the same thing as a | b | c, except that it does not do any redirections. Together with pipe manipulation commands, your example can be implemented as follows:

pout = (pipe)
perr = (pipe)
run-parallel {
  foo > $pout 2> $perr
  pwclose $pout
  pwclose $perr
} {
  bar < $pout
  prclose $pout
} {
  bar2 < $perr
  prclose $perr
}
@krader1961

This comment has been minimized.

krader1961 commented Oct 11, 2017

FWIW, @xiaq, I don't see where any of popen, pipe, prclose, or pwclose are documented. And popen does not appear to be valid. Having said that I am intrigued by the example you wrote which shows that more basic primitives than a | b can be useful in solving complex shell scripting problem. Especially without having to introduce more arcane syntax (something bash/zsh loves to do).

@xiaq

This comment has been minimized.

Member

xiaq commented Oct 11, 2017

@krader1961 Sorry, popen should be pipe, I fixed the original comment.

Good catch for missing doc. I updated the source code but cannot push to prod today, my VPS seems blocked by GFW :-/

@xiaq

This comment has been minimized.

Member

xiaq commented Oct 11, 2017

Updated: https://elvish.io/ref/builtin.html#pipe

I also documented fopen and fclose.

xiaq added a commit that referenced this issue Oct 11, 2017

Add builtin run-parallel.
This addresses #485.
@xiaq

This comment has been minimized.

Member

xiaq commented Oct 11, 2017

I have added a run-parallel builtin, and the sample code I gave above now works. So the functionality is there, albeit with a somewhat verbose syntax, so I am closing this.

For the possibility of extending the pipeline syntax, please discuss in #500.

@xiaq xiaq closed this Oct 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment