Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io/ioutil hangs with too big output from os/exec stderr and stdout pipes #16787

marek-polewski opened this issue Aug 18, 2016 · 1 comment


Copy link

@marek-polewski marek-polewski commented Aug 18, 2016

What version of Go are you using (go version)?

  • 1.6.2
  • 1.6.3
  • 1.7

What operating system and processor architecture are you using (go env)?

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build726085245=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Let's use exec.Command to catch both stdout and stderr.
One of the pipes gets around 82855 chars ( magic number is between 65536 - 131072 ). Now ioutil.ReadAll will hang if I try to read from the second pipe before the first one ( full one ).

Here is the simplified example that only cat a file.
Original code was checking error and there is none of them there:

package main

import (

func main() {
    cmd := exec.Command("cat", "file")
    stdout, _ := cmd.StdoutPipe()
    stderr, _ := cmd.StderrPipe()
    be, _ := ioutil.ReadAll(stderr)
    bo, _ := ioutil.ReadAll(stdout)

    fmt.Printf("%d be size \n", len(be))
    fmt.Printf("%d bo size \n", len(bo))

If order of ReadAll would be opposite, this example will run correctly.

What did you expect to see?
Error or information in docs that those pipes should be handled differently.

What did you see instead?
Binary never exits.

Strace :

[pid 18746] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 18746] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 18746] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 18746] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 18746] futex(0xc420056110, FUTEX_WAKE, 1 <unfinished ...>
[pid 18750] <... futex resumed> )       = 0
[pid 18746] <... futex resumed> )       = 1
[pid 18750] futex(0xc420056110, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 18746] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 18746] futex(0x53fc98, FUTEX_WAIT, 0, {60, 0}

) = -1 ETIMEDOUT (Connection timed out)
[pid 18746] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 18746] futex(0x53fc98, FUTEX_WAIT, 0, {60, 0}


goroutine 1 [syscall, locked to thread]:
syscall.Syscall(0x0, 0x6, 0xc829b06000, 0x200, 0x0, 0xc82685daa0, 0x60)
    /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5, 0xc829b06000, 0x200, 0x200, 0xc829b06200, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x5f
syscall.Read(0x6, 0xc829b06000, 0x200, 0x200, 0xc829b06000, 0x0, 0x0)
    /usr/local/go/src/syscall/syscall_unix.go:161 +0x4d
os.(*File).read(0xc82a0faac8, 0xc829b06000, 0x200, 0x200, 0x7f4700000001, 0x0, 0x0)
    /usr/local/go/src/os/file_unix.go:228 +0x53
os.(*File).Read(0xc82a0faac8, 0xc829b06000, 0x200, 0x200, 0x200, 0x0, 0x0)
    /usr/local/go/src/os/file.go:95 +0x8a
bytes.(*Buffer).ReadFrom(0xc82026e8a0, 0x7f47d95dc1f8, 0xc82a0faac8, 0x0, 0x0, 0x0)
    /usr/local/go/src/bytes/buffer.go:176 +0x23f
io/ioutil.readAll(0x7f47d95dc1f8, 0xc82a0faac8, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/ioutil/ioutil.go:33 +0x156
io/ioutil.ReadAll(0x7f47d95dc1f8, 0xc82a0faac8, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/ioutil/ioutil.go:42 +0x51

This comment has been minimized.

Copy link

@quentinmit quentinmit commented Aug 18, 2016

This is unfortunately just how Unix pipes work. You need to read from both pipes at the same time. What's happening is that cat is trying to write to stdout, but its attempt to write is blocked because the stdout buffer is full. You're trying to ReadAll from stderr, but stderr won't be closed until cat exits, which won't happen until it finishes writing to stdout. So, deadlock.

This is why Command provides Output and CombinedOutput methods; they are careful to always read from both pipes at once. If you want both stdout and stderr but not in the same byte slice, you can also do what CombinedOutput does under the covers and assign separate bytes.Buffer to Stdout and Stderr. Or you can just use Goroutines to read from both at once.

@quentinmit quentinmit closed this Aug 18, 2016
@golang golang locked and limited conversation to collaborators Aug 18, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet
3 participants
You can’t perform that action at this time.