runtime: deadlock when running concurrent builds on MacOS

[sluongng]

What version of Go are you using (go version)?

$ go version
1.20.2

Does this issue reproduce with the latest release?

Yes, this could be reproduce on 1.20.3 but cannot be reproduce on 1.19.8

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/sluongng/Library/Caches/go-build"
GOENV="/Users/sluongng/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/sluongng/work/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/sluongng/work/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/external/go_sdk"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/external/go_sdk/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/sluongng/work/bazelbuild/rules_go/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/ly/gjsqx9wd0cv1ck7xdslrt6t00000gn/T/go-build51901763=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

In Bazel's rules_go, we need to compile a helper binary using go build ... before building anything else.

However, due to different Bazel configurations existing in the same repo (i.e. CGO enable/disable), this same helper binary is built multiple times, once for each unique set of configurations.

When trying to build a test in rules_go with bazel build //tests/core/go_path:copy_path, the helper binary was scheduled by Bazel to be compiled 9 times in parallel. This resulted in 6 different go build processes running in parallel at once.

Each of these build process could spawn multiple compile subprocess as a result

What did you expect to see?

Build to complete without issue

What did you see instead?

All these builds are stuck in a deadlock that completely freeze applications running on the same machine. Opening a new Chrome tab, or running git config would never finish.

Using Activity Monitor's sampling feature we could see call graphs like such for each go build process

It seems like there are 3 types of stack relating to libsystem_kernel.dylib / libsystem_pthread.dylib

    2430 Thread_44978   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2430 ???  (in <unknown binary>)  [0x1358]
    +   2430 runtime.asmcgocall.abi0  (in go) + 124  [0x100b702ac]
    +     2430 runtime.syscall6.abi0  (in go) + 56  [0x100b71a98]
    +       2430 __wait4_nocancel  (in libsystem_kernel.dylib) + 8  [0x1a17a04f4]

    2430 Thread_44999
    + 2430 runtime.asmcgocall.abi0  (in go) + 201  [0x100b702f9]
    +   2430 runtime.pthread_cond_timedwait_relative_np_trampoline.abi0  (in go) + 28  [0x100b717ec]
    +     2430 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1276  [0x1a17d45a0]
    +       2430 __psynch_cvwait  (in libsystem_kernel.dylib) + 8  [0x1a1797710]

    2430 Thread_45038
    + 2430 runtime.kevent_trampoline.abi0  (in go) + 40  [0x100b71518]
    +   2430 kevent  (in libsystem_kernel.dylib) + 8  [0x1a179a060]

The last stack seems to be waiting for some event that never finishes (?).

This bug could be reproduced consistently for me with go 1.20.2 and 1.20.3. It's a bit painful though as I would need to restart my laptop each time. It would go away if I try to build the binary sequentially by limiting Bazel's parallelism, or if I were to downgrade the go version to 1.19.8.

Here is some additional system info

> uname -a
Darwin Sons-Laptop.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

[sluongng]

I tried spindump on the stuck processes, unfortunately, it never finishes as it seems to get stuck on the same IO lock.

[ianlancetaylor]

This seems to be a problem with Bazel building a helper binary. That seems like a problem that needs to be fixed in Bazel. It's not clear to me that there is anything that the Go project can change to fix this.

[sluongng]

Hey Ian, i think that's miss characterizing the issue.

Bazel was mentioned to provide the context that multiple similar "go build" commands were being executed in parallel by a build tool. I don't think Bazel is causing the OS-level lock here. I suspect the root cause is multiple "go build" commands raced against the same kevent api.

[sluongng]

GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-40abec92b10a/bin/external/go_sdk/builder -trimpath
 go/tools/builders/ar.go
 go/tools/builders/asm.go
 go/tools/builders/builder.go
 go/tools/builders/cgo2.go
 go/tools/builders/compilepkg.go
 go/tools/builders/cover.go
 go/tools/builders/edit.go
 go/tools/builders/embedcfg.go
 go/tools/builders/env.go
 go/tools/builders/filter.go
 go/tools/builders/filter_buildid.go
 go/tools/builders/flags.go
 go/tools/builders/generate_nogo_main.go
 go/tools/builders/generate_test_main.go
 go/tools/builders/importcfg.go
 go/tools/builders/link.go
 go/tools/builders/pack.go
 go/tools/builders/read.go
 go/tools/builders/replicate.go
 go/tools/builders/stdlib.go
 go/tools/builders/stdliblist.go
 go/tools/builders/path.go

Here is the effective command that's being executed in parallel by Bazel. The only difference between them would be the output path darwin_arm64-opt-exec-2B5CBBC6-ST-40abec92b10a, everything else is identical.

[ianlancetaylor]

I don't think that Bazel calls go build. I don't see how Bazel could possibly call go build, because the directory layouts are different. And when I look at your screenshots above (by the way, they are very hard to read; please post plain text as plain text, not screenshots), I see external/go_sdk/builder, not go build.

[sluongng]

This is the relevant code from rules_go which will generate the command I shared above. https://github.com/bazelbuild/rules_go/blob/3125d8f0971bee56f069bbb33beb6686109fb71d/go/private/rules/binary.bzl#L451-L463

Here we are compiling the builder binary using go build (in an action called GoToolchainBinaryBuild).
After this, the builder binary will be used as a wrapper for various go tool to compile/link other go packages.

[ianlancetaylor]

OK, and it seems to me that the deadlock occurs when running the builder binary, not when building the builder binary. Am I wrong about that?

[sluongng]

@ianlancetaylor the deadlock does indeed occurs during "building the builder binary".

You could double check my screenshots to see that the stuck process trees are composed of go and sub-process is compile. They correspond to go build and go tool compile respectively.

[ianlancetaylor]

Again, this bug would be much simpler to understand if you used plain text rather than screenshots.

What I see in the first screenshot is "9 actions running" and then 8 lines starting with "GoToolchainBinaryBuild external/go_sdk/builder [for tool]". Nothing in that tells me that it is running "go build" to build the builder binary. That looks to like it is running 8 or 9 instances of the builder binary.

The other screenshots are inconclusive. The Bazel builder binary almost certainly invokes the Go compiler by running go tool compile.

I'm pretty sure that there is an option you can specify to Bazel that tells it to print out the commands that it executes; maybe it is -s. Try doing that to see exactly what it is running when this problem occurs.

Let me add that while of course it is possible that there is a bug in go build, nobody who is not using Bazel is reporting any bug similar to this. So my first guess has to be that this is specific to something that Bazel is doing.

[sluongng]

master ~/work/bazelbuild/rules_go> bazel build tests/core/go_path/... -s
INFO: Analyzed 21 targets (5 packages loaded, 250 targets configured).
INFO: Found 21 targets...
SUBCOMMAND: # //tests/core/go_path/pkg/lib:rename_embedded_src.txt [action 'Copying files', configuration: 0ba77a61754b8ccac68b4c11211b3ebeca24c2f07acb9bc57df3ce73dcecadb5, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
    PATH=/Users/sluongng/Library/Caches/bazelisk/downloads/bazelbuild/bazel-6.1.2-darwin-arm64/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Applications/kitty.app/Contents/MacOS:/Users/sluongng/.fzf/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin \
  /bin/bash -c 'cp -f "$1" "$2"' '' tests/core/go_path/pkg/lib/embedded_src.txt bazel-out/darwin_arm64-fastbuild/bin/tests/core/go_path/pkg/lib/renamed_embedded_src.txt)
# Configuration: 0ba77a61754b8ccac68b4c11211b3ebeca24c2f07acb9bc57df3ce73dcecadb5
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //tests/core/go_path/pkg/lib:rename_embedded_src.txt [action 'Copying files', configuration: bf4310556fb43dee678e82527598f2b62aa15dd9e2440952a54d52ac073c8b8b, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
    PATH=/Users/sluongng/Library/Caches/bazelisk/downloads/bazelbuild/bazel-6.1.2-darwin-arm64/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Applications/kitty.app/Contents/MacOS:/Users/sluongng/.fzf/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin \
  /bin/bash -c 'cp -f "$1" "$2"' '' tests/core/go_path/pkg/lib/embedded_src.txt bazel-out/darwin_arm64-fastbuild-ST-6b5761d5db88/bin/tests/core/go_path/pkg/lib/renamed_embedded_src.txt)
# Configuration: bf4310556fb43dee678e82527598f2b62aa15dd9e2440952a54d52ac073c8b8b
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 810285c14034f59b4ec014927bb4d3c6e157d8e61c8effc5da7e00c82a3cc87b, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-a72284a01f96/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 810285c14034f59b4ec014927bb4d3c6e157d8e61c8effc5da7e00c82a3cc87b
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 9a20c4fb1fb26d87e7032264c4c8d169122a2a6e0fd5df83853a733162459093, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-40abec92b10a/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 9a20c4fb1fb26d87e7032264c4c8d169122a2a6e0fd5df83853a733162459093
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: cc61e454e8918ef0674fd1fac3c89c2953d19080940582aa1be83b3dbd3d9ea6, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-be3721871959/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: cc61e454e8918ef0674fd1fac3c89c2953d19080940582aa1be83b3dbd3d9ea6
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: a3e486db8af89926670f15fd2989125c5cbb16b83f3198f0aef6a01f7954d5cb, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-e846b08c7501/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: a3e486db8af89926670f15fd2989125c5cbb16b83f3198f0aef6a01f7954d5cb
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 28cc2ea5e1b2f72029b4f077a078b277812d3f857d77b15be60bfee00bfd4696, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 28cc2ea5e1b2f72029b4f077a078b277812d3f857d77b15be60bfee00bfd4696
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 7ddfe315b68e233265bdcbf0bdae3786330900349c4b5bc224e8023d4ff0017b, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-b94cd9d981ea/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 7ddfe315b68e233265bdcbf0bdae3786330900349c4b5bc224e8023d4ff0017b
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 7020bcec75482155add832dcb68bf1c0a40860576dc3d263e32492270922fd88, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-0097a518a0b7/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 7020bcec75482155add832dcb68bf1c0a40860576dc3d263e32492270922fd88
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 264349405e4d117e3b07dae9deb33599b67f77a7f017dc8ef8cdf62738143375, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-99e44bd3b10b/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 264349405e4d117e3b07dae9deb33599b67f77a7f017dc8ef8cdf62738143375
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 24916cb83c3cd56e4590ab5c07a3d7a97b3cd556454f53c61bbccdbfc4dbf23c, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-ffe6de5a5d6b/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 24916cb83c3cd56e4590ab5c07a3d7a97b3cd556454f53c61bbccdbfc4dbf23c
# Execution platform: @local_config_platform//:host
[36 / 104] 9 actions running
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox ...

Hey Ian,

I do apologise for the lack of text.
This issue is quite painful to reproduce as everytime the deadlock happen, I have to reset my laptop as most process/application stopped working.

The above is the --subcommand you are looking for.