runtime: deadlock when running concurrent builds on MacOS #59657

sluongng · 2023-04-16T15:16:01Z

What version of Go are you using (`go version`)?

$ go version
1.20.2

Does this issue reproduce with the latest release?

Yes, this could be reproduce on 1.20.3 but cannot be reproduce on 1.19.8

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/sluongng/Library/Caches/go-build"
GOENV="/Users/sluongng/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/sluongng/work/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/sluongng/work/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/external/go_sdk"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/external/go_sdk/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/sluongng/work/bazelbuild/rules_go/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/ly/gjsqx9wd0cv1ck7xdslrt6t00000gn/T/go-build51901763=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

In Bazel's rules_go, we need to compile a helper binary using go build ... before building anything else.

However, due to different Bazel configurations existing in the same repo (i.e. CGO enable/disable), this same helper binary is built multiple times, once for each unique set of configurations.

When trying to build a test in rules_go with bazel build //tests/core/go_path:copy_path, the helper binary was scheduled by Bazel to be compiled 9 times in parallel. This resulted in 6 different go build processes running in parallel at once.

Each of these build process could spawn multiple compile subprocess as a result

What did you expect to see?

Build to complete without issue

What did you see instead?

All these builds are stuck in a deadlock that completely freeze applications running on the same machine. Opening a new Chrome tab, or running git config would never finish.

Using Activity Monitor's sampling feature we could see call graphs like such for each go build process

It seems like there are 3 types of stack relating to libsystem_kernel.dylib / libsystem_pthread.dylib

    2430 Thread_44978   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2430 ???  (in <unknown binary>)  [0x1358]
    +   2430 runtime.asmcgocall.abi0  (in go) + 124  [0x100b702ac]
    +     2430 runtime.syscall6.abi0  (in go) + 56  [0x100b71a98]
    +       2430 __wait4_nocancel  (in libsystem_kernel.dylib) + 8  [0x1a17a04f4]

    2430 Thread_44999
    + 2430 runtime.asmcgocall.abi0  (in go) + 201  [0x100b702f9]
    +   2430 runtime.pthread_cond_timedwait_relative_np_trampoline.abi0  (in go) + 28  [0x100b717ec]
    +     2430 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1276  [0x1a17d45a0]
    +       2430 __psynch_cvwait  (in libsystem_kernel.dylib) + 8  [0x1a1797710]

    2430 Thread_45038
    + 2430 runtime.kevent_trampoline.abi0  (in go) + 40  [0x100b71518]
    +   2430 kevent  (in libsystem_kernel.dylib) + 8  [0x1a179a060]

The last stack seems to be waiting for some event that never finishes (?).

This bug could be reproduced consistently for me with go 1.20.2 and 1.20.3. It's a bit painful though as I would need to restart my laptop each time. It would go away if I try to build the binary sequentially by limiting Bazel's parallelism, or if I were to downgrade the go version to 1.19.8.

Here is some additional system info

> uname -a
Darwin Sons-Laptop.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

The text was updated successfully, but these errors were encountered:

sluongng · 2023-04-16T15:18:27Z

I tried spindump on the stuck processes, unfortunately, it never finishes as it seems to get stuck on the same IO lock.

ianlancetaylor · 2023-04-17T00:59:13Z

This seems to be a problem with Bazel building a helper binary. That seems like a problem that needs to be fixed in Bazel. It's not clear to me that there is anything that the Go project can change to fix this.

sluongng · 2023-04-17T09:52:11Z

Hey Ian, i think that's miss characterizing the issue.

Bazel was mentioned to provide the context that multiple similar "go build" commands were being executed in parallel by a build tool. I don't think Bazel is causing the OS-level lock here. I suspect the root cause is multiple "go build" commands raced against the same kevent api.

sluongng · 2023-04-17T11:27:46Z

GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-40abec92b10a/bin/external/go_sdk/builder -trimpath
 go/tools/builders/ar.go
 go/tools/builders/asm.go
 go/tools/builders/builder.go
 go/tools/builders/cgo2.go
 go/tools/builders/compilepkg.go
 go/tools/builders/cover.go
 go/tools/builders/edit.go
 go/tools/builders/embedcfg.go
 go/tools/builders/env.go
 go/tools/builders/filter.go
 go/tools/builders/filter_buildid.go
 go/tools/builders/flags.go
 go/tools/builders/generate_nogo_main.go
 go/tools/builders/generate_test_main.go
 go/tools/builders/importcfg.go
 go/tools/builders/link.go
 go/tools/builders/pack.go
 go/tools/builders/read.go
 go/tools/builders/replicate.go
 go/tools/builders/stdlib.go
 go/tools/builders/stdliblist.go
 go/tools/builders/path.go

Here is the effective command that's being executed in parallel by Bazel. The only difference between them would be the output path darwin_arm64-opt-exec-2B5CBBC6-ST-40abec92b10a, everything else is identical.

ianlancetaylor · 2023-04-18T00:30:20Z

I don't think that Bazel calls go build. I don't see how Bazel could possibly call go build, because the directory layouts are different. And when I look at your screenshots above (by the way, they are very hard to read; please post plain text as plain text, not screenshots), I see external/go_sdk/builder, not go build.

sluongng · 2023-04-18T08:36:29Z

This is the relevant code from rules_go which will generate the command I shared above. https://github.com/bazelbuild/rules_go/blob/3125d8f0971bee56f069bbb33beb6686109fb71d/go/private/rules/binary.bzl#L451-L463

Here we are compiling the builder binary using go build (in an action called GoToolchainBinaryBuild).
After this, the builder binary will be used as a wrapper for various go tool to compile/link other go packages.

ianlancetaylor · 2023-04-18T17:21:44Z

OK, and it seems to me that the deadlock occurs when running the builder binary, not when building the builder binary. Am I wrong about that?

sluongng · 2023-04-18T17:30:34Z

@ianlancetaylor the deadlock does indeed occurs during "building the builder binary".

You could double check my screenshots to see that the stuck process trees are composed of go and sub-process is compile. They correspond to go build and go tool compile respectively.

ianlancetaylor · 2023-04-18T20:12:33Z

Again, this bug would be much simpler to understand if you used plain text rather than screenshots.

What I see in the first screenshot is "9 actions running" and then 8 lines starting with "GoToolchainBinaryBuild external/go_sdk/builder [for tool]". Nothing in that tells me that it is running "go build" to build the builder binary. That looks to like it is running 8 or 9 instances of the builder binary.

The other screenshots are inconclusive. The Bazel builder binary almost certainly invokes the Go compiler by running go tool compile.

I'm pretty sure that there is an option you can specify to Bazel that tells it to print out the commands that it executes; maybe it is -s. Try doing that to see exactly what it is running when this problem occurs.

Let me add that while of course it is possible that there is a bug in go build, nobody who is not using Bazel is reporting any bug similar to this. So my first guess has to be that this is specific to something that Bazel is doing.

sluongng · 2023-04-19T07:16:41Z

master ~/work/bazelbuild/rules_go> bazel build tests/core/go_path/... -s
INFO: Analyzed 21 targets (5 packages loaded, 250 targets configured).
INFO: Found 21 targets...
SUBCOMMAND: # //tests/core/go_path/pkg/lib:rename_embedded_src.txt [action 'Copying files', configuration: 0ba77a61754b8ccac68b4c11211b3ebeca24c2f07acb9bc57df3ce73dcecadb5, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
    PATH=/Users/sluongng/Library/Caches/bazelisk/downloads/bazelbuild/bazel-6.1.2-darwin-arm64/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Applications/kitty.app/Contents/MacOS:/Users/sluongng/.fzf/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin \
  /bin/bash -c 'cp -f "$1" "$2"' '' tests/core/go_path/pkg/lib/embedded_src.txt bazel-out/darwin_arm64-fastbuild/bin/tests/core/go_path/pkg/lib/renamed_embedded_src.txt)
# Configuration: 0ba77a61754b8ccac68b4c11211b3ebeca24c2f07acb9bc57df3ce73dcecadb5
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //tests/core/go_path/pkg/lib:rename_embedded_src.txt [action 'Copying files', configuration: bf4310556fb43dee678e82527598f2b62aa15dd9e2440952a54d52ac073c8b8b, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
    PATH=/Users/sluongng/Library/Caches/bazelisk/downloads/bazelbuild/bazel-6.1.2-darwin-arm64/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/sluongng/bin:/opt/homebrew/opt/openjdk@17/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Applications/kitty.app/Contents/MacOS:/Users/sluongng/.fzf/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin:/snap/bin:/Users/sluongng/.local/bin:/usr/local/go/bin:/Users/sluongng/work/golang/bin:/Users/sluongng/.cargo/bin:/Users/sluongng/.rvm/bin \
  /bin/bash -c 'cp -f "$1" "$2"' '' tests/core/go_path/pkg/lib/embedded_src.txt bazel-out/darwin_arm64-fastbuild-ST-6b5761d5db88/bin/tests/core/go_path/pkg/lib/renamed_embedded_src.txt)
# Configuration: bf4310556fb43dee678e82527598f2b62aa15dd9e2440952a54d52ac073c8b8b
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 810285c14034f59b4ec014927bb4d3c6e157d8e61c8effc5da7e00c82a3cc87b, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-a72284a01f96/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 810285c14034f59b4ec014927bb4d3c6e157d8e61c8effc5da7e00c82a3cc87b
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 9a20c4fb1fb26d87e7032264c4c8d169122a2a6e0fd5df83853a733162459093, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-40abec92b10a/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 9a20c4fb1fb26d87e7032264c4c8d169122a2a6e0fd5df83853a733162459093
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: cc61e454e8918ef0674fd1fac3c89c2953d19080940582aa1be83b3dbd3d9ea6, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-be3721871959/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: cc61e454e8918ef0674fd1fac3c89c2953d19080940582aa1be83b3dbd3d9ea6
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: a3e486db8af89926670f15fd2989125c5cbb16b83f3198f0aef6a01f7954d5cb, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-e846b08c7501/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: a3e486db8af89926670f15fd2989125c5cbb16b83f3198f0aef6a01f7954d5cb
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 28cc2ea5e1b2f72029b4f077a078b277812d3f857d77b15be60bfee00bfd4696, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 28cc2ea5e1b2f72029b4f077a078b277812d3f857d77b15be60bfee00bfd4696
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 7ddfe315b68e233265bdcbf0bdae3786330900349c4b5bc224e8023d4ff0017b, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-b94cd9d981ea/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 7ddfe315b68e233265bdcbf0bdae3786330900349c4b5bc224e8023d4ff0017b
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 7020bcec75482155add832dcb68bf1c0a40860576dc3d263e32492270922fd88, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-0097a518a0b7/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 7020bcec75482155add832dcb68bf1c0a40860576dc3d263e32492270922fd88
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 264349405e4d117e3b07dae9deb33599b67f77a7f017dc8ef8cdf62738143375, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-99e44bd3b10b/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 264349405e4d117e3b07dae9deb33599b67f77a7f017dc8ef8cdf62738143375
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 24916cb83c3cd56e4590ab5c07a3d7a97b3cd556454f53c61bbccdbfc4dbf23c, execution platform: @local_config_platform//:host]
(cd /private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/execroot/io_bazel_rules_go && \
  exec env - \
  /bin/bash -c 'GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-ffe6de5a5d6b/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 24916cb83c3cd56e4590ab5c07a3d7a97b3cd556454f53c61bbccdbfc4dbf23c
# Execution platform: @local_config_platform//:host
[36 / 104] 9 actions running
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox
    GoToolchainBinaryBuild external/go_sdk/builder [for tool]; 3s darwin-sandbox ...

Hey Ian,

I do apologise for the lack of text.
This issue is quite painful to reproduce as everytime the deadlock happen, I have to reset my laptop as most process/application stopped working.

The above is the --subcommand you are looking for.

ianlancetaylor · 2023-04-19T21:40:15Z

Thanks. You're quite right, it does seem to be running go build. It appears to be running the same command several times in parallel. Any idea why it is doing that?

The stack traces you show above seem to show cmd/go waiting for a subcommand to complete. Based on the process tree it is likely waiting for cmd/compile to compile something.

What does top show when this is happening? Are the cmd/compile processes working, or are they waiting for something? I can't think of what they might be waiting for.

sluongng · 2023-04-20T09:32:09Z

Thanks. You're quite right, it does seem to be running go build. It appears to be running the same command several times in parallel. Any idea why it is doing that?

It's because of different Bazel's build configurations, generated from a combination of multiple transitions augmented the Bazel buildgraph. So each action is executed multiple times, once for each unique build configuration.

What does top show when this is happening? Are the cmd/compile processes working, or are they waiting for something? I can't think of what they might be waiting for.

Im a bit busy until the end of this week so I will provide a reproduce with top later.

But from my original issue description, you could review the sampling result of the go processes, which seem to be stuck on some runtime.kevent_tramboline calls. Not sure if top could provide additional info or if there are some other ways 🤔

Note: Since this issue was created, we have merged bazel-contrib/rules_go#3536 in rules_go to limit the concurrency of the go build action, making the deadlock less likely to happen, as a temporary workaround. So reproduce attempt should use an earlier commit in rules_go for a more consistent result.

ianlancetaylor · 2023-04-21T23:10:15Z

The kevent_trampoline calls are normal and don't mean anything special. The key in the stack trace is that the program is waiting at a call to wait4. In other words, it's waiting for a subprocess to complete. Given that it is running go build, and given the process tree that you showed, it is most likely waiting for go tool compile to complete. So if there is a deadlock we need to understand why go tool compile is not completing.

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 16, 2023

sluongng mentioned this issue Apr 17, 2023

go_tool_binary: set GOMAXPROCS to 1 bazel-contrib/rules_go#3536

Merged

mknyszek added this to Go Compiler / Runtime Apr 19, 2023

mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 19, 2023

mknyszek added this to the Backlog milestone Apr 19, 2023

mknyszek moved this to Todo in Go Compiler / Runtime Apr 19, 2023

bcmills mentioned this issue Nov 3, 2023

os,internal/poll: reads from closed pipes sometimes do not unblock on darwin #63937

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: deadlock when running concurrent builds on MacOS #59657

runtime: deadlock when running concurrent builds on MacOS #59657

sluongng commented Apr 16, 2023 •

edited

Loading

sluongng commented Apr 16, 2023

ianlancetaylor commented Apr 17, 2023

sluongng commented Apr 17, 2023

sluongng commented Apr 17, 2023

ianlancetaylor commented Apr 18, 2023

sluongng commented Apr 18, 2023

ianlancetaylor commented Apr 18, 2023

sluongng commented Apr 18, 2023

ianlancetaylor commented Apr 18, 2023

sluongng commented Apr 19, 2023

ianlancetaylor commented Apr 19, 2023

sluongng commented Apr 20, 2023

ianlancetaylor commented Apr 21, 2023

runtime: deadlock when running concurrent builds on MacOS #59657

runtime: deadlock when running concurrent builds on MacOS #59657

Comments

sluongng commented Apr 16, 2023 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

sluongng commented Apr 16, 2023

ianlancetaylor commented Apr 17, 2023

sluongng commented Apr 17, 2023

sluongng commented Apr 17, 2023

ianlancetaylor commented Apr 18, 2023

sluongng commented Apr 18, 2023

ianlancetaylor commented Apr 18, 2023

sluongng commented Apr 18, 2023

ianlancetaylor commented Apr 18, 2023

sluongng commented Apr 19, 2023

ianlancetaylor commented Apr 19, 2023

sluongng commented Apr 20, 2023

ianlancetaylor commented Apr 21, 2023

sluongng commented Apr 16, 2023 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?