os/exec: sometimes Run doesn't return even though the process has finished

What version of Go are you using (go version)?

$ go version
go version go1.20.2 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/xxx/.cache/go-build"
GOENV="/home/xxx/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/xxx/gocode/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/xxx/gocode"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/home/xxx/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/home/xxx/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1084134332=/tmp/go-build -gno-record-gcc-switches"

What did you do?

a.go:

package main

import (
	"fmt"
	"io"
	"os"
	"os/exec"
	"time"
)

func main() {
	cmd := exec.Command("./b")
	pr, pw := io.Pipe()
	_ = pr
	cmd.Stdout = pw
	cmd.Stderr = os.Stderr
	go func() {
		<-time.After(5 * time.Second)
		if err := cmd.Process.Signal(os.Interrupt); err != nil {
			fmt.Println("Signal:", err)
		}
		time.Sleep(time.Second)
		if err := cmd.Process.Kill(); err != nil {
			fmt.Println("Kill:", err)
		}
	}()
	if err := cmd.Run(); err != nil {
		fmt.Println("Run[a]:", err)
	}
	fmt.Println("Run returned.")
}

b.go:

package main

import (
	"fmt"
	"io"
	"os"
	"os/exec"
)

func main() {
	cmd := exec.Command("tail", "-f", "/dev/null")
	pr, pw := io.Pipe()
	_ = pw
	cmd.Stdin = pr
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	if err := cmd.Run(); err != nil {
		fmt.Println("Run[b]:", err)
	}
}

After building the binaries and putting them in the same directory, I run ./a.

What did you expect to see?

The program prints Run returned. in 5 seconds and exits.

What did you see instead?

Kill: os: process already finished

The program doesn't exit until I press Ctrl+C.

[cherrymui]

The documentation of Cmd.Stdout https://pkg.go.dev/os/exec#Cmd.Stdout mentions

	// Otherwise, during the execution of the command a separate goroutine
	// reads from the process over a pipe and delivers that data to the
	// corresponding Writer. In this case, Wait does not complete until the
	// goroutine reaches EOF or encounters an error or a nonzero WaitDelay
	// expires.

In particular, Wait does not complete until EOF from stdout.

In your case, in a, a pipe is connected to cmd.Stdout (where the cmd is b), which is b's os.Stdout, which in b is connected to cmd.Stdout (where the cmd is tail). So, a's cmd.Stdout pipe is eventually connected to tail's stdout.

When a terminates b with a signal, b does not send a signal to kill tail, nor does it close tail's stdin, so tail is still running and its stdout is still open. Therefore, in a, cmd.Stdout has not reached EOF and so Wait does not complete, as documented. Run uses Wait to wait for the process exit, so it also does not complete.

If either in a you don't connect cmd.Stdout to a pipe, or in b you don't connect cmd.Stdout to os.Stdout, Run will return.

Closing as working as intended. Thanks.

[ianlancetaylor]

Note that Go 1.20 introduces a new WaitDelay field to help address this kind of problem. https://pkg.go.dev/os/exec#Cmd.WaitDelay

Thanks, WaitDelay works like a charm. It's just surprising that Wait should block on input from a grandchild process.