Skip to content

Commit

Permalink
fix: set parallelism to match Linux container CPU (#3901)
Browse files Browse the repository at this point in the history
<!--

Hi, thanks for contributing!

Please make sure you read our CONTRIBUTING guide.

Also, add tests and the respective documentation changes as well.

-->

Currently Goreleaser uses `runtime.NumCPU()` as the default value if
`--parallelism` is not set.
However, this will get the number of CPUs on the host even when
Goreleaser is run in a container with a limit on the maximum number of
CPUs that can be used (typically in a Kubernetes pod).
Actually, `docker run --cpus=1 goreleaser/goreleaser --debug` shows
`parallelism: 4` on my machine.
This behavior causes CPU throttling, which increases execution time and,
in the worst case, terminates with an error.
I ran into this problem with Jenkins where the agent runs on pod
([Kubernetes plugin for
Jenkins](https://plugins.jenkins.io/kubernetes/)).

This commit introduces
[automaxprocs](https://github.com/uber-go/automaxprocs) to fix this
issue.
This library sets `GOMAXPROCS` to match Linux container CPU quota.
I have also looked for a library that can get CPU quota more directly,
but this seems to be the best I could find.
The reason it is set in a different notation from the automaxprocs
README is to prevent logs from being displayed
([comment](uber-go/automaxprocs#18 (comment))).

I would have liked to write a test, but this change is dependent on the
number of CPUs in the execution environment, so I could not.
Instead, I wrote a Dockerfile for testing

```Dockerfile
FROM golang:1.20.2

WORKDIR /go/app
RUN sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b /usr/local/bin
COPY . .
RUN task build
```

and confirmed built binary shows expected parallelism by following
commands:

```sh
docker build --file Dockerfile.test . -t test-goreleaser
docker run --cpus=1 test-goreleaser ./goreleaser build --snapshot --debug # parallelism: 1
docker run test-goreleaser ./goreleaser build --snapshot --debug # parallelism: 4
```

I also ran the built binary on my Macbook and it was fine.
  • Loading branch information
marukaz committed Apr 2, 2023
1 parent d524d93 commit b495c90
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 2 deletions.
2 changes: 1 addition & 1 deletion cmd/build.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ func setupPipeline(ctx *context.Context, options buildOpts) []pipeline.Piper {

func setupBuildContext(ctx *context.Context, options buildOpts) error {
ctx.Deprecated = options.deprecated // test only
ctx.Parallelism = runtime.NumCPU()
ctx.Parallelism = runtime.GOMAXPROCS(0)
if options.parallelism > 0 {
ctx.Parallelism = options.parallelism
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/release.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ func releaseProject(options releaseOpts) (*context.Context, error) {

func setupReleaseContext(ctx *context.Context, options releaseOpts) {
ctx.Deprecated = options.deprecated // test only
ctx.Parallelism = runtime.NumCPU()
ctx.Parallelism = runtime.GOMAXPROCS(0)
if options.parallelism > 0 {
ctx.Parallelism = options.parallelism
}
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,7 @@ require (
gitlab.com/digitalxero/go-conventional-commit v1.0.7 // indirect
go.mongodb.org/mongo-driver v1.11.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.uber.org/automaxprocs v1.5.2
golang.org/x/exp v0.0.0-20230124195608-d38c7dcee874 // indirect
golang.org/x/mod v0.9.0 // indirect
golang.org/x/net v0.8.0 // indirect
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -2229,6 +2229,8 @@ go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/automaxprocs v1.5.1/go.mod h1:BF4eumQw0P9GtnuxxovUd06vwm1o18oMzFtK66vU6XU=
go.uber.org/automaxprocs v1.5.2 h1:2LxUOGiR3O6tw8ui5sZa2LAaHnsviZdVOUZw4fvbnME=
go.uber.org/automaxprocs v1.5.2/go.mod h1:eRbA25aqJrxAbsLO0xy5jVwPt7FQnRgjW+efnwa1WM0=
go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
go.uber.org/goleak v1.1.11-0.20210813005559-691160354723/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
go.uber.org/goleak v1.1.11/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
Expand Down
7 changes: 7 additions & 0 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ import (
"runtime"
"runtime/debug"

"github.com/caarlos0/log"
"github.com/charmbracelet/lipgloss"
"github.com/goreleaser/goreleaser/cmd"
"github.com/muesli/termenv"
"go.uber.org/automaxprocs/maxprocs"
)

// nolint: gochecknoglobals
Expand All @@ -24,6 +26,11 @@ func init() {
if os.Getenv("CI") != "" {
lipgloss.SetColorProfile(termenv.TrueColor)
}
// automatically set GOMAXPROCS to match available CPUs.
// GOMAXPROCS will be used as the default value for the --parallelism flag.
if _, err := maxprocs.Set(); err != nil {
log.WithError(err).Fatal("failed to set GOMAXPROCS")
}
}

func main() {
Expand Down

0 comments on commit b495c90

Please sign in to comment.