Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: too many open files in system #58788

Closed
maxgiraldo opened this issue Feb 28, 2023 · 11 comments
Closed

cmd/go: too many open files in system #58788

maxgiraldo opened this issue Feb 28, 2023 · 11 comments
Labels
GoCommand cmd/go OS-Darwin WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@maxgiraldo
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.20.1 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/maxgiraldo/Library/Caches/go-build"
GOENV="/Users/maxgiraldo/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/maxgiraldo/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/maxgiraldo/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/maxgiraldo/go/src/github.com/maxgiraldo/test_server/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lf/r3_cvxl10n502833_y2v3rr80000gn/T/go-build2784566378=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Trying to run basic server:

package main

import (
    "fmt"
    "net/http"
)

func hello(w http.ResponseWriter, req *http.Request) {

    fmt.Fprintf(w, "hello\n")
}

func headers(w http.ResponseWriter, req *http.Request) {

    for name, headers := range req.Header {
        for _, h := range headers {
            fmt.Fprintf(w, "%v: %v\n", name, h)
        }
    }
}

func main() {

    http.HandleFunc("/hello", hello)
    http.HandleFunc("/headers", headers)

    http.ListenAndServe(":8090", nil)
}

I am not able to run or install the program.

What did you expect to see?

The server running.

What did you see instead?

/usr/local/go/src/vendor/golang.org/x/text/unicode/bidi/core.go:9:2: open /usr/local/go/src/log/example_test.go: too many open files in system

Why is go trying to open these files?

@ianlancetaylor
Copy link
Contributor

What is the exact command that you run that prints that error message?

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 28, 2023
@maxgiraldo
Copy link
Author

maxgiraldo commented Feb 28, 2023

Commands I ran that produce same error:

go run .
go run main.go

@maxgiraldo
Copy link
Author

maxgiraldo commented Feb 28, 2023

I ran go mod tidy earlier and it completely messed up everything and failed because it tried to search my entire filesystem. Not sure if causal relationship is there, but that context might help.

I also tried uninstalling and reinstalling go and my terminal but that didn’t help.

@ianlancetaylor
Copy link
Contributor

What does go run -x print?

@ianlancetaylor ianlancetaylor changed the title net/http: too many open files in system cmd/go: too many open files in system Feb 28, 2023
@ianlancetaylor ianlancetaylor added the GoCommand cmd/go label Feb 28, 2023
@maxgiraldo
Copy link
Author

WORK=/var/folders/lf/r3_cvxl10n502833_y2v3rr80000gn/T/go-build2569235747
/usr/local/go/src/vendor/golang.org/x/text/unicode/bidi/core.go:9:2: open /usr/local/go/src/log/example_test.go: too many open files in system

@bcmills
Copy link
Member

bcmills commented Feb 28, 2023

/usr/local/go/src/vendor/golang.org/x/text/unicode/bidi/core.go:9:2: open /usr/local/go/src/log/example_test.go: too many open files in system

Why is go trying to open these files?

golang.org/x/text/unicode/bidi imports the log package.

Compiling the log package itself shouldnt require opening example_test.go, but as of Go 1.19 the go` command indexes package information and uses that instead of reopening files for each build. That is supposed to happen only once per build, but of course if that fails the index won't be written to the cache.

But, that seems like something of a red herring — if the error is too many open files in system, what else on your system is holding files open? (If it were just the go command, the error would presumably be too many open files instead.)

A good starting point for that investigation might be to run lsof and see if you spot a pattern.

@bcmills bcmills added OS-Darwin WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Feb 28, 2023
@maxgiraldo
Copy link
Author

maxgiraldo commented Mar 1, 2023

I have about 8007 open files in my system on fresh reboot and closing my applications. I set a soft limit for 10240, but there's a hard limit of unlimited. Even if I set the soft limit to unlimited, go still throws the same error.

ulimit -aS
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1333
-n: file descriptors                10240

I also tried changing the maxfile limit for MacOS following these instructions, which changes the default from 256 to 64000 when you open a fresh terminal (instead of having to manually set it per session), but that had no effect:

ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       2000
-n: file descriptors                64000

So I think I addressed the "too many open files in system" issue, but Go is still complaining. I suspect this is some MacOS-specific ARM64 weirdness and may just be unique to my machine.

@maxgiraldo maxgiraldo reopened this Mar 1, 2023
@bcmills
Copy link
Member

bcmills commented Mar 1, 2023

FWIW, based on https://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-files-in-system-error-on-os-x-10-7-1 I checked a couple of the builders operated by the Go project and found:

darwin-amd64-13-0:

$ lsof | wc -l
    3311
$ sysctl kern.maxfiles
kern.maxfiles: 30720
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 10240

darwin-arm64-12:

$ lsof | wc -l
    3643
$ sysctl kern.maxfiles
kern.maxfiles: 122880
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 61440

Even on the amd64 builder with a maxfilesperproc limit of only 10240, that seems like plenty of headroom. I would be surprised if any of those limits were adjusted when the builder images were created. (There aren't any such steps listed in https://cs.opensource.google/go/x/build/+/master:env/darwin/setup-notes.md.)

Perhaps you have some kind of third-party service installed that is holding a lot of files open?

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Mar 1, 2023
@maxgiraldo
Copy link
Author

Yeah, i’m going to do a clean install and see if I can reproduce.

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Mar 1, 2023
@earthboundkid
Copy link
Contributor

I was running out of file descriptors on Mac OS every few days a few months ago. It started around the time I installed Tailscale's Golink program, and it seemed to be related to that because, it seemed to stop when I stopped running it, and since I upgraded Golink a few weeks ago, the problem hasn't recurred. I'm still unsure what the root cause was though, maybe that's a coincidence. I've also had instability with my Time Machine backups.

@maxgiraldo
Copy link
Author

maxgiraldo commented Mar 2, 2023

I erased all contents and settings on my Mac and re-installed go and was able to run it. There's still about ~6265 open files by default on Ventura 13.1, but go no longer complains. I guess I'll be careful about what third-party software I install next time.

And thanks to everyone for responding so quickly and jumping on this. Really impressed by the community and support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go OS-Darwin WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

5 participants