os/exec: cmd.Output/StartProcess hangs on macOS 12.3

[dprotaso]

What version of Go are you using (go version)?

$ go version
go1.17.8 darwin/amd64
and 
go version go1.18 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

OS: macOS Monterey Version 12.3 (21E230)
Arch: amd64 (Intel i9)
Bash: GNU bash, version 5.1.16(1)-release (x86_64-apple-darwin20.6.0)
KO_DOCKER_REPO set to a gcr repo - but I've reproduced this using kind.local as well (see kind setup)
Terminal: iTerm2 Build 3.4.15

go env Output

$ go env
go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dprotasowski/Library/Caches/go-build"
GOENV="/Users/dprotasowski/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/dprotasowski/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/dprotasowski/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.18/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.18/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/dprotasowski/work/serving4/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/xs/xfp1z4cn643c46w69lv8bv8w0000gp/T/go-build2573651288=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

$ go install github.com/google/ko@v0.11.2
$ git clone https://github.com/knative/serving
$ cd serving
# default shell is zsh run bash 5.1
$ bash
# Run the following a bunch of times until it hangs ~20 tries at times
$ ko resolve -R -f config/core/ > /dev/null

What did you expect to see?

The ko program uses the os/exec.Cmd package to kick off go builds and perform registry credential lookups.
The program runs to completion

What did you see instead?

The program hangs calling cmd.Output(). It looks like it a fork/exec fails as ps -l it shows the forked process but it looks like exec wasn't invoked as the process name is the same.

  UID   PID  PPID        F CPU PRI NI       SZ    RSS WCHAN     S             ADDR TTY           TIME CMD
  502 19733 19731     4006   0  31  0 34584196   3808 -      S                   0 ttys001    0:00.06 -zsh
  502 21014 19733     4006   0  31  0 34551188   9672 -      S+                  0 ttys001    0:09.78 watch -n1 ps | grep ko
  502 20937 20935     4006   0  31  0 34752132   4132 -      S                   0 ttys002    0:00.15 -zsh
  502 63498 63496     4006   0  31  0 34599668   4316 -      S                   0 ttys003    0:00.07 -zsh
  502 63608 63498     4006   0  31  0 36285588  44612 -      S+                  0 ttys003    0:01.95 nvim .
  502 21101 21099     4006   0  31  0 34731652   3884 -      S                   0 ttys005    0:00.07 -zsh
  502 23766 21101     4006   0  31  0 34293012   2452 -      S                   0 ttys005    0:00.05 bash
  502 53232 23766     4006   0  31  0 35033184 110996 -      S+                  0 ttys005    0:00.76 ko resolve -R -f config/core/
  502 53344 53232        6   0  31  0 34901180    372 -      R+                  0 ttys005    1:35.16 ko resolve -R -f config/core/

After some time the process state was

  UID   PID  PPID        F CPU PRI NI       SZ    RSS WCHAN     S             ADDR TTY           TIME CMD
  502 55784 55782     4006   0  31  0 34739844   3876 -      S                   0 ttys000    0:00.06 -zsh
  502 21014 19733     4006   0  31  0 34551188   9672 -      S+                  0 ttys001    0:10.63 watch -n1 ps | grep ko
  502 20937 20935     4006   0  31  0 34752132   4132 -      S                   0 ttys002    0:00.16 -zsh
  502 63498 63496     4006   0  31  0 34599668   4316 -      S                   0 ttys003    0:00.07 -zsh
  502 63608 63498     4006   0  31  0 36285588  44612 -      S+                  0 ttys003    0:01.95 nvim .
  502 21101 21099     4006   0  31  0 34731652   3884 -      S                   0 ttys005    0:00.07 -zsh
  502 23766 21101     4006   0  31  0 34293012   2452 -      S                   0 ttys005    0:00.05 bash
  502 53232 23766     4006   0  31  0 35033188 111004 -      S+                  0 ttys005    0:00.79 ko resolve -R -f config/core/
  502 53344 56663      806   0  31  0 34901184    744 -      SX+                 0 ttys005    7:45.97 ko resolve -R -f config/core/

Parent Process Stack Dumps

Go routine stack traces for the parent and child process are here: https://gist.github.com/dprotaso/1e1c4c2abea19dbfbdbb61b221c0fdcc

Notes on reproducibility

I wasn't able to reproduce this in zsh only bash

Using Apple's Terminal.app I couldn't reproduce the issue (~10 runs). Comparing the difference between the terminal apps I noticed iTerm had some extra env vars.

I figured out an env var having an affect. __CF_USER_TEXT_ENCODING. Setting this did cause ko in Apple's Terminal.app to start hanging

export __CF_USER_TEXT_ENCODING=0x1F6:0x0:0x52

After unset'ing that env var in iTerm I didn't encounter a hang (20 tries)

Unsure how that envvar has an affect on fork/exec

[mknyszek]

It's possible #52226 is related. It's also a hang on darwin/amd64. At least one user has reported that the hang is in exec, as you point out.

[dprotaso]

Yeah I'm just surprised how I'm able to reproduce this bug