os/exec: failures with "netpollBreak write failed" on linux-amd64 since 2021-11-10

[bcmills]

greplogs --dashboard -md -l -e 'fatal error: runtime: netpollBreak write failed'

2021-11-11T04:54:05-4b27d40/linux-amd64-race
2021-11-10T21:32:50-f410786/linux-amd64-fedora
(Note the 2-year gap and difference in platforms here! This looks like a regression.)
2019-11-04T23:41:34-383b447/darwin-386-10_14
2019-11-04T16:32:38-7dcd343/darwin-amd64-nocgo
2019-11-02T21:51:21-177a36a/dragonfly-amd64
2019-11-02T21:51:14-a3ffb0d/darwin-amd64-10_14
2019-11-02T21:51:07-40b7455/darwin-amd64-nocgo
2019-11-02T05:52:33-8de0bb7/netbsd-amd64-8_0
2019-11-01T21:41:41-9bde9b4/darwin-amd64-10_14
2019-11-01T05:38:51-e96fd13/darwin-amd64-10_14

[bcmills]

(@prattmic, @ChrisHines: Is this possibly related to #44343?)

[prattmic]

errno 32 is EPIPE, meaning the other end of the pipe (netpollBreakRd) is closed. The runtime never closes that FD, so I suspect this is a bug of something closing an FD it doesn't own.

[bcmills]

The runtime never closes that FD, so I suspect this is a bug of something closing an FD it doesn't own.

Ooh, that sounds like another os.File finalizer race, maybe! Or, looks like the failure is in os/exec in both of the above — could be related to #49500?

[bcmills]

Yep, here it is, I think:

go/src/os/exec/exec_test.go

Lines 52 to 67 in 4d8db00

	// We have no good portable way to check whether an FD is open.
	// We use NewFile to create a *os.File, which lets us
	// know whether it is open, but then we have to cope with
	// the finalizer on the *os.File.
	f := os.NewFile(fd, "")
	if _, err := f.Stat(); err != nil {
	// Close the file to clear the finalizer.
	// We expect the Close to fail.
	f.Close()
	} else {
	fmt.Printf("fd %d open at test start\n", fd)
	haveUnexpectedFDs = true
	// Use a global variable to avoid running
	// the finalizer, which would close the FD.
	unfinalizedFiles = append(unfinalizedFiles, f)
	}

[prattmic]

Oof, that is quite broken. But are we sure that is the cause of these crashes? That only runs for GO_WANT_HELPER_PROCESS=1, but the test logs look like it is the main test process crashing.

Either way, this may be another case of the new pacer shaking out a bug with slightly earlier GC runs.

cc @mknyszek