io: Copy to a pipe prevents process exit (Go 1.24rc2 on Linux regression)

[kolyshkin]

Go version

go version go1.24rc2 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/kir/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/kir/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build957638587=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/kir/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/kir/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/kir/sdk/go1.24rc2'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/kir/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/kir/sdk/go1.24rc2/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24rc2'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

There is a regression in Go 1.24rc2 (git-bisect points to 1) caused by a (not yet confirmed) Linux kernel bug with sendmail(2)/splice(2), which I just reported 2.

In short, when sendfile(2) or splice(2) is used to copy data to a pipe, and another process is having the other end of this file, this prevents that other process from exit. You can get a short C repro from 2, and here's a Go repro:

package main

import (
	"io"
	"log"
	"os"
	"os/exec"
)

func main() {
	// Create a pipe for stdin
	r, w, err := os.Pipe()
	if err != nil {
		log.Fatal(err)
	}

	// Create a short-lived process (like ps) with stdin connected to pipe
	cmd := exec.Command("ps")
	cmd.Stdin = r
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	if err := cmd.Start(); err != nil {
		log.Fatal(err)
	}
	log.Print("Run child pid=", cmd.Process.Pid)

	// Close read end in parent.
	r.Close()

	// Copy from stdin to pipe - this should trigger sendfile in Go 1.24.
	go func() {
		_, err = io.Copy(w, os.Stdin)
		if err != nil {
			log.Fatal("io.Copy: ", err)
		}
	}()

	if err := cmd.Wait(); err != nil {
		log.Fatal("wait: ", err)
	}
}

What did you see happen?

When the above repro is run with go1.24rc2, the process is not exiting:

[kir@kir-tp1 sendfile-vs-pipe]$ go1.24rc2 run repro.go 
2025/01/21 18:32:54 Run child pid=2180908
    PID TTY          TIME CMD
  63304 pts/1    00:00:03 bash
2180738 pts/1    00:00:00 go1.24rc2
2180747 pts/1    00:00:00 go
2180902 pts/1    00:00:00 repro
2180908 pts/1    00:00:00 ps

(and the process hangs here).

NOTE if you can't repro this, you probably pressed Enter in the terminal. Do not do anything with this terminal!.

In a different terminal, you can check the status of the child process (using the pid from the above output):

[kir@kir-tp1 linux]$ ps -f -p 2180908
UID          PID    PPID  C STIME TTY          TIME CMD
kir      2180908 2180902  0 18:32 pts/1    00:00:00 [ps]

As you can see, ps thinks that the process is a kernel thread. This happens because the process is half-exited and some /proc/$PID/ entries (root, cwd, `exe) are no longer valid, like it is with kernel threads.

Now, to see where the process is stuck:

[kir@kir-tp1 linux]$ sudo cat /proc/2180908/stack
[<0>] pipe_release+0x1f/0x100
[<0>] __fput+0xde/0x2a0
[<0>] task_work_run+0x59/0x90
[<0>] do_exit+0x309/0xab0
[<0>] do_group_exit+0x30/0x80
[<0>] __x64_sys_exit_group+0x18/0x20
[<0>] x64_sys_call+0x14b4/0x14c0
[<0>] do_syscall_64+0x82/0x160
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

What did you expect to see?

No stuck process.

With an older Go version, everything works as it should:

[kir@kir-tp1 sendfile-vs-pipe]$ go1.23.4 run repro.go 
2025/01/21 18:38:16 Run child pid=2181265
    PID TTY          TIME CMD
  63304 pts/1    00:00:03 bash
2181065 pts/1    00:00:00 go1.23.4
2181071 pts/1    00:00:00 go
2181259 pts/1    00:00:00 repro
2181265 pts/1    00:00:00 ps
[kir@kir-tp1 sendfile-vs-pipe]$ 

[kolyshkin]

First, let's wait for kernel folks to confirm this.

Thinking about a workaround, my best solution for now is to not use sendfile(2) on Linux when writing to a pipe.

[gabyhelp]

Related Issues

• os/exec: possible race handling stdout&stderr pipes #69060
• execution hangs when I try to open named pipe file #58854 (closed)
• Linux: open namepipe file may cause process hang #33050 (closed)
• io/ioutil hangs with too big output from os/exec stderr and stdout pipes #16787 (closed)
• os/exec: Current requirement for all reads to be completed before calling cmd.Wait can block indefinitely #60309 (closed)
• os/exec: syscall.forkExec hang when spawning multiple processes concurrently on `darwin` #61080 (closed)
• affected/package: os/exec, context not work. #53700 (closed)
• os/exec: cmd.Start got a deadlock when set cmd.Stdin #47061 (closed)
• os/exec: CombinedOutput hangs when exec'd task forks a "daemon" #13155 (closed)
• os/exec: sometimes Run doesn't return even though the process has finished #59055 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

cc @golang/runtime

[prattmic]

Thanks for testing the RC and the great bug report!

I am also curious to see what the kernel folks have to say. My immediate thought was that exit_group should never block indefinitely and anything causing it to do so is clearly a kernel bug. That said, man 2 _exit says "_exit() does close open file descriptors, and this may cause an unknown delay, waiting for pending output to finish." So that does not agree with my initial interpretation.

Regardless, even if it is considered a kernel bug, we'll have to work around it for older kernels. It seems that the obvious options are:

1. Revert https://go.dev/cl/603295.
2. Add a special exception to avoid using sendfile just with pipe FDs.

Given how late we are in the release freeze, I'd lean towards (1) as the less risky option, and then we could consider applying a new version of the CL for 1.25.

cc @panjf2000