cmd/go: "mod tidy" exhausts file descriptors, crashes other macOS apps

[Handzo]

(Note: this issue is about go mod tidy. For gopls mcp exhausting fds, see #77814. --adonovan)
(See also #77808, a dup of the current issue with more information. --adonovan)

gopls version

Build info
----------
golang.org/x/tools/gopls v0.17.1
    golang.org/x/tools/gopls@v0.17.1 h1:Mt/DSfnnSe3dyf6MH/dZZ0iww+viHNhAFc4rEYDiOAw=
    github.com/BurntSushi/toml@v1.4.1-0.20240526193622-a339e1f7089c h1:pxW6RcqyfI9/kWtOwnv/G+AzdKuy2ZrqINhenH4HyNs=
    github.com/google/go-cmp@v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
    golang.org/x/exp/typeparams@v0.0.0-20231108232855-2478ac86f678 h1:1P7xPZEwZMoBoz0Yze5Nx2/4pxj6nw9ZqHWXqP0iRgQ=
    golang.org/x/mod@v0.22.0 h1:D4nJWe9zXqHOmWqj4VMOJhvzj7bEZg4wEYa759z1pH4=
    golang.org/x/sync@v0.9.0 h1:fEo0HyrW1GIgZdpbhCRO0PkJajUS5H9IFUztCgEo2jQ=
    golang.org/x/telemetry@v0.0.0-20241106142447-58a1122356f5 h1:TCDqnvbBsFapViksHcHySl/sW4+rTGNIAoJJesHRuMM=
    golang.org/x/text@v0.20.0 h1:gK/Kv2otX8gz+wn7Rmb3vT96ZwuoxnQlY+HlJVj7Qug=
    golang.org/x/tools@v0.27.1-0.20241219162658-575221bfbda3 h1:kgwdasJRsdDWYgWcEgMF424DiXwwXHSb3V8xVTi//i8=
    golang.org/x/vuln@v1.0.4 h1:SP0mPeg2PmGCu03V+61EcQiOjmpri2XijexKdzv8Z1I=
    honnef.co/go/tools@v0.5.1 h1:4bH5o3b5ZULQ4UrBmP+63W9r7qIkqJClEA9ko5YKx+I=
    mvdan.cc/gofumpt@v0.7.0 h1:bg91ttqXmi9y2xawvkuMXyvAA/1ZGJqYAEGjXuP0JXU=
    mvdan.cc/xurls/v2@v2.5.0 h1:lyBNOm8Wo71UknhUs4QTFUNNMyxy2JEIaKKo0RWOh+8=
go: go1.23.6

go env

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/handzo/Library/Caches/go-build'
GOENV='/Users/handzo/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/handzo/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/handzo/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/handzo/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/handzo/work/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/c_/xr46l7b17tl1vx8ynq0wqmjdjbq0ht/T/go-build3884281298=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Opening project in emacs with lsp-mode.

What did you see happen?

in ~= 30 minutes can do nothing. Getting "Too many open files in system" on any operation.

What did you expect to see?

Everything works well

Editor and settings

directory filters: ["-directory1", "-directory2", ...],
file watch ignored directories: ["directory1", "directory2", ...]

Logs

$ lsof -n | awk '{print $1}' | uniq -c | sort -rn | head -n 1
11417 go

$ ps -ax | grep go
2975 ??         1:34.37 go mod tidy -modfile=/var/folders/c_/xr46l7b17tl1vx8ynq0wqmjdjbq0ht/T/gopls-tempmod2565361955/go.mod

$ lsof -p 2975 | wc -l
10124

[gabyhelp]

Related Issues

• x/tools/gopls: too many memory usage #37759 (closed)
• x/tools/gopls: memory issues / multiple instances of gopls running while using neovim/vim-go/coc.vim #41998 (closed)
• x/tools/gopls: gopls seems to be leaking memory on macOS Big Sur #42342 (closed)
• x/tools/gopls: high memory consumption when used with a monorepo #37670 (closed)
• x/tools/gopls: emacs crash when using gopls with Go 1.20 #58236 (closed)
• x/tools/gopls: panic and crash in golang.org/x/telemetry/internal/counter #64577 (closed)
• x/tools/gopls: long initial workspace load durations for workspace with many packages #69523
• x/tools/cmd/gopls: fails on filepaths with spaces #34270 (closed)
• x/tools/gopls: gopls walks entire filesystem of a go module #52993 (closed)
• x/tools/gopls: gopls stopped responding when gomod failed. #38818 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[adonovan]

@samthanawalla @matloob Any idea why the go mod tidy command might have 11417 open files? If there's some parallel recursion over the file tree, we should impose a reasonable limit on parallel I/O.

[matloob]

Usually when we recurse over the file tree we use par.Queues that limit the number of operations to GOMAXPROCS, and the opened files for each operation should be closed at the end of those operations.

One thing that comes to mind that may or may not be the issue is the module index: we mmap files that index each of the modules, and each of the packages in the main/work module and we don't close those files for the lifetime of the go command so it's possible that in a very large project could open too many of those files? I'm curious how many packages go list all ./... in the root of the main/work module returns, and how many entries go list -m all returns?

[adonovan]

Does the go command close the fd for each mapped file? If so then the mapping shouldn't continue to count against the fd limit.

[gopherbot]

Change https://go.dev/cl/652435 mentions this issue: cmd/go/internal/modindex: clean modroot and pkgdir for openIndexPackage