Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: mod tidy with ~1M subdirectories takes a very long time and then crashes computer #59860

Closed
anacrolix opened this issue Apr 27, 2023 · 12 comments
Labels
GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin Unfortunate WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@anacrolix
Copy link
Contributor

This probably seems a bit bizarre, and comes out of #56494 (comment) and #55955 (comment). I'm not sure how it's possible, I think it's a MacOS specific issue, possibly to do with ulimit -n, which I understand may have been changed around go1.19.

Testing it is painful as my computer actually crashes as a result, and the project is private, but I'm happy to what I can to expose the issue or make a reproduction.

What version of Go are you using (go version)?

go version devel go1.21-e007c515f3 Mon Apr 24 15:49:38 2023 +0000 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/anacrolix/Library/Caches/go-build"
GOENV="/Users/anacrolix/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/anacrolix/go/pkg/mod"
GONOPROXY="github.com/anacrolix,github.com/some-company"
GONOSUMDB="github.com/anacrolix,github.com/some-company"
GOOS="darwin"
GOPATH="/Users/anacrolix/go"
GOPRIVATE="github.com/anacrolix,github.com/some-company"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/anacrolix/src/go.master"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/anacrolix/src/go.master/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="devel go1.21-e007c515f3 Mon Apr 24 15:49:38 2023 +0000"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/abs/path/to/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/m4/f67w9zfx1pl386_2yjs7xtkm0000gn/T/go-build2760210553=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

go mod tidy

What did you expect to see?

Nothing. Maybe a tidier go.mod and go.sum.

What did you see instead?

CPU is at 300-400% and memory climbs to ~400-500MiB over a few minutes then my computer switches off.

@seankhliao
Copy link
Member

pass in -v -x flags?

@seankhliao seankhliao changed the title go mod tidy takes a very long time and then crashes computer cmd/go: mod tidy takes a very long time and then crashes computer Apr 27, 2023
@anacrolix
Copy link
Contributor Author

anacrolix commented Apr 27, 2023

It only happens when the system has been running for a few hours.

After it restarts it shows

panic(cpu 2 caller 0xfffffe00236407b0): watchdog timeout: no checkins from watchdogd in 92 seconds (8454 total checkins since monitoring last enabled)

While the issue occurs, I can't start new applications, and I have one application that shows "xcrun too many open files".

There was no output from go mod tidy, even with -x -v.

@anacrolix
Copy link
Contributor Author

It has something to do with having massive numbers of subdirectories and no hard limit on file descriptors. If I run with ulimit -nH 1024, I actually get output from go mod tidy -x -v. My go-build cache has 1.6m entries, and my project has 500k files in a lot of sub directories, and go mod tidy -x -v is outputting lots of go: finding module for package project-import-path/some/sub/directory.

@bcmills bcmills self-assigned this Apr 27, 2023
@bcmills bcmills added this to the Backlog milestone Apr 27, 2023
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin labels Apr 27, 2023
@bcmills
Copy link
Contributor

bcmills commented Apr 27, 2023

my project has 500k files in a lot of sub directories

Do each of those subdirectories contain at least one .go file, or are some or most of them non-Go?

If you set GODEBUG=goindex=0 in your environment, does that change the behavior? (I wonder if this is related to #54226.)

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 27, 2023
@bcmills
Copy link
Contributor

bcmills commented Apr 27, 2023

CPU is at 300-400%

Perhaps send SIGQUIT to the go command once it has been running for a while and send me the stack dump?

@anacrolix
Copy link
Contributor Author

I have a smaller reproduction:

cd `mktemp -d`
go mod init shit
for a in `seq 1000000`; do mkdir "$a"; done
go mod tidy

I don't know if this causes issues on platforms other than MacOS.

@bcmills
Copy link
Contributor

bcmills commented May 1, 2023

I can reproduce high CPU usage on Linux, but it doesn't cause a crash — and a module with that many directory entries seems well outside of the limits of what Go can reasonably be expected to support anyway. (I see ~3.9GiB of directory entries alone!)

$ go version
go version devel go1.21-71ad46cd Fri Apr 28 01:33:31 2023 +0000 linux/amd64

$ go mod init example
go: creating new go.mod: module example

$ for a in `seq 1000000`; do mkdir "$a"; done

$ go mod tidy
go: warning: "all" matched no packages

$ du -hs .
3.9G    .

So, my suggestion is to either escalate this to Apple as a macOS kernel bug, or find a way to structure your project so that either you have fewer directory entries overall or they are excluded from go mod tidy (for example, by placing them in a parent directory beginning with _ or . or placing a go.mod file in a parent directory to prune it out of the module).

@bcmills bcmills removed their assignment May 1, 2023
@bcmills bcmills modified the milestones: Backlog, Unplanned May 1, 2023
@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels May 1, 2023
@bcmills bcmills changed the title cmd/go: mod tidy takes a very long time and then crashes computer cmd/go: mod tidy with ~1M subdirectories takes a very long time and then crashes computer May 1, 2023
@anacrolix
Copy link
Contributor Author

I think having ulimit -nH prevents the crash, so I think there's value in having a reasonable limit in place to prevent the worst case, even if the conditions for it to occur are fairly extreme.

anacrolix added a commit to getlantern/replica that referenced this issue May 23, 2023
Moved to https://github.com/getlantern/replica-scripts. This is because the large number of working files for bucket-sql causes the go tool to have issues. golang/go#59860
@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jul 9, 2023
@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 9, 2023
@anacrolix
Copy link
Contributor Author

This is a mistake.

@bcmills
Copy link
Contributor

bcmills commented Aug 9, 2023

The crash seems to be platform-specific, and we have no reason to believe that the platform-specific crash is a Go bug. Again, I suggest that you escalate it to Apple as a potential macOS kernel bug.

The solution to the performance issue seems straightforward: find a different way to structure the project.
(Also note that #42965 would provide a mechanism for ignoring specific root directories, which could perhaps give you a way to keep go mod tidy from scanning subtrees if these aren't all in the module root.)

@anacrolix
Copy link
Contributor Author

Thanks @bcmills, I think #42965 might work as a solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin Unfortunate WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

4 participants