Skip to content

Give special handling to the first stage's stdin to prevent a deadlock #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 12, 2023

Conversation

mhagger
Copy link
Member

@mhagger mhagger commented Dec 4, 2023

The pipe's stdin needs to be wrapped with something like an io.NopCloser() to prevent the pipe from closing it.

We used to use io.NopCloser() for this purpose, but it has a subtle problem. If the first stage is a Command, then we wants to set the exec.Cmd's Stdin to an io.Reader corresponding to p.stdin. If Cmd.Stdin is an *os.File, then the file descriptor can be passed to the subcommand directly; there is no need for this process to create a pipe and copy the data into the input side of the pipe. But if p.stdin is not an *os.File, then this optimization is prevented. And even worse, it also has the side effect that the goroutine that copies from Cmd.Stdin into the pipe doesn't terminate until that fd is closed by the writing side.

That isn't always what we want. Consider, for example, the following snippet, where the subcommand's stdin is set to the stdin of the enclosing Go program, but wrapped with io.NopCloser:

cmd := exec.Command("ls")
cmd.Stdin = io.NopCloser(os.Stdin)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Run()

In this case, we don't want the Go program to wait for os.Stdin to close (because ls isn't even trying to read from its stdin). But it does: exec.Cmd doesn't recognize that Cmd.Stdin is an *os.File, so it sets up a pipe and copies the data itself, and this goroutine doesn't terminate until cmd.Stdin (i.e., the Go program's own stdin) is closed. But if, for example, the Go program is run from an interactive shell session, that might never happen, in which case the program will fail to terminate, even after ls exits.

So instead, in this special case, wrap p.stdin in our own nopCloser, which behaves like io.NopCloser, except that pipe.CommandStage knows how to unwrap it before passing it to exec.Cmd.

/cc @elhmn

Add a test of a pipeline whose stage is a command that never reads
from stdin, but where stdin is never closed by the caller. This should
succeed but it doesn't.
See the long comment in `Pipeline.Start()` for more information. This
fixes the failing test that was introduced in the previous commit.
This test is similar to `TestPipelineStdinFileThatIsNeverClosed()`
except that stdin is not a naked `*os.File` but rather is wrapped to
disguise it from `os.Cmd`. This test currently deadlocks, so it is
marked `Skip()` (but we should figure out a way to get it to timeout,
at a minimum.
@mhagger mhagger requested a review from a team as a code owner December 4, 2023 11:32
@mhagger mhagger merged commit f3fb7f1 into main Dec 12, 2023
@mhagger mhagger deleted the unwrap-stdin branch December 12, 2023 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants