cmd/go: go list does not scale with GOMAXPROCS #63136

goto1134 · 2023-09-21T07:45:17Z

What version of Go are you using (`go version`)?

master ace1494

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH='arm64'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOOS='darwin'

What did you do?

Clone Kubernetes
Run GOMAXPROCS=10 go list -json -m -u -mod=readonly -debug-trace=trace-default.txt all
Run GOMAXPROCS=20 go list -json -m -u -mod=readonly -debug-trace=trace-maxproc.txt all

The second command takes the same amount of time as the first one.
Here are the collected traces:

What did you expect to see?

The second command runs faster and scales, depending on the GOMAXPROCS value.

What did you see instead?

The second comment took the same amount of time as the first one.

Here are the collected traces:
trace-default.txt
trace-maxproc20.txt

From the traces, it's evident that the bottleneck is in getting the modules in a loop here:
https://github.com/golang/go/blob/ace1494d9235be94f1325ab6e45105a446b3224c/src/cmd/go/internal/modload/list.go#L277C1-L286C4

The text was updated successfully, but these errors were encountered:

Fixes golang#63136

gopherbot · 2023-09-21T07:58:39Z

Change https://go.dev/cl/530037 mentions this issue: cmd/go: scale go list with GOMAXPROCS

Fixes golang#63136

dany74q · 2023-10-17T08:53:31Z

Any update on this one ? It makes life much better for large multi-module mono-repos

goto1134 · 2023-10-17T10:11:37Z

@dany74q , the next step is to provide benchmarks. I want to do it sometime later. Probably in November.

I hope this change will be accepted for the February release of Go.

goos: darwin goarch: arm64 pkg: cmd/go/internal/modload/list_test │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 98.71µ ± 5% 101.54µ ± 2% ~ (p=0.143 n=10) ListModules/Cmd-10 273.8m ± 1% 109.7m ± 1% -59.93% (p=0.000 n=10) ListModules/K8S-10 12.199 ± 3% 5.416 ± 3% -55.60% (p=0.000 n=10) geomean 69.08m 39.22m -43.23% Fixes golang#63136

Fixes golang#63136

goos: darwin goarch: arm64 pkg: cmd/go/internal/modload/list_test │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 98.71µ ± 5% 101.54µ ± 2% ~ (p=0.143 n=10) ListModules/Cmd-10 273.8m ± 1% 109.7m ± 1% -59.93% (p=0.000 n=10) ListModules/K8S-10 12.199 ± 3% 5.416 ± 3% -55.60% (p=0.000 n=10) geomean 69.08m 39.22m -43.23% Fixes golang#63136

goos: darwin goarch: arm64 pkg: cmd/go/internal/modload/list_test │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 124.0µ ± 2% 125.7µ ± 1% ~ (p=0.063 n=10) ListModules/Cmd-10 403.3m ± 0% 109.7m ± 1% -72.80% (p=0.000 n=10) ListModules/K8S-10 18.261 ± 0% 5.406 ± 2% -70.39% (p=0.000 n=10) geomean 97.01m 42.09m -56.61% Fixes golang#63136

AdallomRoy · 2024-04-16T12:16:33Z

@goto1134 any updates on this?

goto1134 · 2024-04-16T21:29:26Z

@AdallomRoy, the PR is still being reviewed and waiting for me to revisit the performance tests after the latest review comments. It's my next task on the list 🙂

Fixes golang#63136

goos: darwin goarch: arm64 pkg: cmd/go cpu: Apple M1 Max │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 6.414m ± 0% 6.154m ± 3% -4.06% (p=0.000 n=20) ListModules/Cmd-10 823.8m ± 3% 296.6m ± 1% -63.99% (p=0.000 n=20) ListModules/K8S-10 37.075 ± 3% 9.003 ± 3% -75.72% (p=0.000 n=20) geomean 580.8m 254.2m -56.22% │ old.txt │ new.txt │ │ sys-sec/op │ sys-sec/op vs base │ ListModules/Empty-10 2.065m ± 1% 1.985m ± 3% -3.86% (p=0.000 n=20) ListModules/Cmd-10 43.48m ± 5% 33.32m ± 6% -23.37% (p=0.000 n=20) ListModules/K8S-10 2.106 ± 9% 1.798 ± 5% -14.64% (p=0.000 n=20) geomean 57.40m 49.18m -14.33% │ old.txt │ new.txt │ │ user-sec/op │ user-sec/op vs base │ ListModules/Empty-10 2.459m ± 0% 2.419m ± 2% -1.63% (p=0.000 n=20) ListModules/Cmd-10 33.55m ± 3% 26.73m ± 8% -20.30% (p=0.000 n=20) ListModules/K8S-10 1106.9m ± 11% 834.4m ± 7% -24.62% (p=0.000 n=20) geomean 45.03m 37.79m -16.08% Fixes golang#63136

AdallomRoy · 2024-05-21T07:47:29Z

@goto1134 any updates on this? it's been another month and performance is still abysmal.

goto1134 · 2024-05-21T17:04:51Z

@goto1134 any updates on this? it's been another month and performance is still abysmal.

Hey, @AdallomRoy ! As you might see, I provided all the changes requested by reviewers. Could you clarify how I can help you?

Fixes golang#63136

goos: darwin goarch: arm64 pkg: cmd/go cpu: Apple M1 Max │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 7.695m ± 10% 7.562m ± 9% ~ (p=0.678 n=20) ListModules/Cmd-10 339.7m ± 3% 112.9m ± 2% -66.76% (p=0.000 n=20) ListModules/K8S-10 13.657 ± 10% 3.511 ± 3% -74.29% (p=0.000 n=20) geomean 329.3m 144.2m -56.21% │ old.txt │ new.txt │ │ sys-sec/op │ sys-sec/op vs base │ ListModules/Empty-10 2.523m ± 16% 2.297m ± 12% ~ (p=0.904 n=20) ListModules/Cmd-10 70.64m ± 20% 46.33m ± 9% -34.41% (p=0.000 n=20) ListModules/K8S-10 2.552 ± 6% 2.541 ± 9% ~ (p=0.883 n=20) geomean 76.90m 64.66m -15.91% │ old.txt │ new.txt │ │ user-sec/op │ user-sec/op vs base │ ListModules/Empty-10 3.068m ± 7% 2.967m ± 4% ~ (p=1.000 n=20) ListModules/Cmd-10 21.72m ± 12% 14.92m ± 2% -31.31% (p=0.000 n=20) ListModules/K8S-10 604.7m ± 9% 523.1m ± 8% -13.49% (p=0.000 n=20) geomean 34.28m 28.50m -16.87% Fixes golang#63136

goos: darwin goarch: arm64 pkg: cmd/go cpu: Apple M1 Max │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 7.768m ± 5% 7.768m ± 6% ~ (p=0.989 n=20) ListModules/Cmd-10 272.3m ± 2% 137.8m ± 2% -49.40% (p=0.000 n=20) ListModules/K8S-10 10.741 ± 2% 2.525 ± 5% -76.49% (p=0.000 n=20) geomean 283.2m 139.3m -50.82% │ old.txt │ new.txt │ │ sys-sec/op │ sys-sec/op vs base │ ListModules/Empty-10 2.380m ± 9% 2.443m ± 9% ~ (p=0.314 n=20) ListModules/Cmd-10 51.84m ± 13% 47.27m ± 14% ~ (p=0.289 n=20) ListModules/K8S-10 1.660 ± 8% 1.485 ± 28% ~ (p=0.512 n=20) geomean 58.95m 55.56m -5.75% │ old.txt │ new.txt │ │ user-sec/op │ user-sec/op vs base │ ListModules/Empty-10 3.034m ± 4% 3.053m ± 3% ~ (p=0.445 n=20) ListModules/Cmd-10 18.01m ± 11% 15.39m ± 5% -14.55% (p=0.000 n=20) ListModules/K8S-10 407.6m ± 11% 209.2m ± 49% -48.67% (p=0.000 n=20) geomean 28.13m 21.42m -23.86% Fixes golang#63136

Benchmark from the go-list-benchmark branch shows the following result: goos: darwin goarch: arm64 pkg: cmd/go cpu: Apple M1 Max │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 7.768m ± 5% 7.768m ± 6% ~ (p=0.989 n=20) ListModules/Cmd-10 272.3m ± 2% 137.8m ± 2% -49.40% (p=0.000 n=20) ListModules/K8S-10 10.741 ± 2% 2.525 ± 5% -76.49% (p=0.000 n=20) geomean 283.2m 139.3m -50.82% │ old.txt │ new.txt │ │ sys-sec/op │ sys-sec/op vs base │ ListModules/Empty-10 2.380m ± 9% 2.443m ± 9% ~ (p=0.314 n=20) ListModules/Cmd-10 51.84m ± 13% 47.27m ± 14% ~ (p=0.289 n=20) ListModules/K8S-10 1.660 ± 8% 1.485 ± 28% ~ (p=0.512 n=20) geomean 58.95m 55.56m -5.75% │ old.txt │ new.txt │ │ user-sec/op │ user-sec/op vs base │ ListModules/Empty-10 3.034m ± 4% 3.053m ± 3% ~ (p=0.445 n=20) ListModules/Cmd-10 18.01m ± 11% 15.39m ± 5% -14.55% (p=0.000 n=20) ListModules/K8S-10 407.6m ± 11% 209.2m ± 49% -48.67% (p=0.000 n=20) geomean 28.13m 21.42m -23.86% Fixes golang#63136

Updates golang#63136

gopherbot · 2024-08-24T08:12:51Z

Change https://go.dev/cl/608135 mentions this issue: cmd/go: add benchmark for go list -m

Updates golang#63136

Benchmark from the go-list-benchmark branch shows the following result: goos: darwin goarch: arm64 pkg: cmd/go cpu: Apple M1 Max │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ ListModules/Empty-10 7.768m ± 5% 7.768m ± 6% ~ (p=0.989 n=20) ListModules/Cmd-10 272.3m ± 2% 137.8m ± 2% -49.40% (p=0.000 n=20) ListModules/K8S-10 10.741 ± 2% 2.525 ± 5% -76.49% (p=0.000 n=20) geomean 283.2m 139.3m -50.82% │ old.txt │ new.txt │ │ sys-sec/op │ sys-sec/op vs base │ ListModules/Empty-10 2.380m ± 9% 2.443m ± 9% ~ (p=0.314 n=20) ListModules/Cmd-10 51.84m ± 13% 47.27m ± 14% ~ (p=0.289 n=20) ListModules/K8S-10 1.660 ± 8% 1.485 ± 28% ~ (p=0.512 n=20) geomean 58.95m 55.56m -5.75% │ old.txt │ new.txt │ │ user-sec/op │ user-sec/op vs base │ ListModules/Empty-10 3.034m ± 4% 3.053m ± 3% ~ (p=0.445 n=20) ListModules/Cmd-10 18.01m ± 11% 15.39m ± 5% -14.55% (p=0.000 n=20) ListModules/K8S-10 407.6m ± 11% 209.2m ± 49% -48.67% (p=0.000 n=20) geomean 28.13m 21.42m -23.86% Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Sep 21, 2023

cmd/go: scale go list with GOMAXPROCS

8289881

Fixes golang#63136

goto1134 mentioned this issue Sep 21, 2023

cmd/go: scale go list with GOMAXPROCS #63137

Closed

bcmills added ToolSpeed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go labels Sep 21, 2023

bcmills added this to the Backlog milestone Sep 21, 2023

goto1134 changed the title ~~cmd/go: go list does not scale with GOMAXPROC~~ cmd/go: go list does not scale with GOMAXPROCS Sep 21, 2023

goto1134 added a commit to goto1134/go that referenced this issue Sep 21, 2023

cmd/go: removes a mutex and preserves module order

dfaef9b

Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Jan 9, 2024

cmd/go: scale go list with GOMAXPROCS

ec6602f

Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Jan 9, 2024

cmd/go: removes a mutex and preserves module order

d8d4832

Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Apr 30, 2024

cmd/go: scale go list with GOMAXPROCS

87c3ea2

Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Apr 30, 2024

cmd/go: removes a mutex and preserves module order

563f626

Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Aug 16, 2024

cmd/go: removes a mutex and preserves module order

453dddd

Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Aug 16, 2024

cmd/go: replace a semaphore with a queue

f65ca70

Fixes golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Aug 24, 2024

cmd/go: add benchmark for go list -m

e9044a1

Updates golang#63136

goto1134 mentioned this issue Aug 24, 2024

cmd/go: add benchmark for go list -m #69052

Open

goto1134 added a commit to goto1134/go that referenced this issue Aug 24, 2024

cmd/go: removes redundant writeFile function

121ef98

Updates golang#63136

goto1134 added a commit to goto1134/go that referenced this issue Aug 24, 2024

cmd/go: remove redundant writeFile function

2b864be

Updates golang#63136

gopherbot closed this as completed in 7f05a25 Aug 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: go list does not scale with GOMAXPROCS #63136

cmd/go: go list does not scale with GOMAXPROCS #63136

goto1134 commented Sep 21, 2023 •

edited

Loading

gopherbot commented Sep 21, 2023

dany74q commented Oct 17, 2023

goto1134 commented Oct 17, 2023

AdallomRoy commented Apr 16, 2024

goto1134 commented Apr 16, 2024

AdallomRoy commented May 21, 2024

goto1134 commented May 21, 2024 •

edited

Loading

gopherbot commented Aug 24, 2024

cmd/go: go list does not scale with GOMAXPROCS #63136

cmd/go: go list does not scale with GOMAXPROCS #63136

Comments

goto1134 commented Sep 21, 2023 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

gopherbot commented Sep 21, 2023

dany74q commented Oct 17, 2023

goto1134 commented Oct 17, 2023

AdallomRoy commented Apr 16, 2024

goto1134 commented Apr 16, 2024

AdallomRoy commented May 21, 2024

goto1134 commented May 21, 2024 • edited Loading

gopherbot commented Aug 24, 2024

goto1134 commented Sep 21, 2023 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

goto1134 commented May 21, 2024 •

edited

Loading