Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: limit the number of files open at the same time #21550

Closed
chenzhongtao opened this issue Aug 22, 2017 · 6 comments

Comments

Projects
None yet
5 participants
@chenzhongtao
Copy link
Contributor

commented Aug 22, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.9rc2

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH="amd64"
GOHOSTOS="linux"

What did you do?

src/cmd/compile/internal/gc/noder.go : func parseFiles(filenames []string) uint
If the length of filenames is large, it will open too many files at
the same time, it will cause EMFILE error. modify code to limit max
goroutines.

@dcheney-atlassian

This comment has been minimized.

Copy link

commented Aug 22, 2017

Hi,

Thanks for opening this issue. Thank you for describing where the bug occurs, but can you please tell us how you trigger it. Output from go build / go test / go install would be best.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 22, 2017

Change https://golang.org/cl/57751 mentions this issue: cmd/compile: limit the number of files open at the same time

@chenzhongtao

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

if your Go packages with enough source files more than ulimit -n, it can trigger it.
you can try by :

export GOROOT_BOOTSTRAP=/usr/lib/go-go1.9rc2
ulimit -n 32
cd /home/go/src
./all.bash

the output maybe like this:

bootstrap/cmd/compile/internal/ssa

src/bootstrap/cmd/compile/internal/ssa/decompose.go:0:0: open src/bootstrap/cmd/compile/internal/ssa/decompose.go: too many open files

this problem can be avioded if you modify max open files of system.

but in my compiling environment, my code in cifs filesystem(linux mount filesystem from windows), in this environment , concurrent open the file will lead to error: input/output error,

go build myproject

golang.org/x/sys/unix

../common/src/golang.org/x/sys/unix/sockcmsg_linux.go:0:0: open ../common/src/golang.org/x/sys/unix/sockcmsg_linux.go: input/output error

so I hope I can control the max open files by environment variable GOMAXPROCS.
the modify is :

sem := make(chan struct{}, runtime.GOMAXPROCS(0)) // Semaphore to guard against parsing too many source files simultaneously

export GOMAXPROCS=1
go build myproject // it can avoid error in my environment

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2017

Can you please explain why you need to build the Go distribution with such a small ulimit.

@chenzhongtao

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2017

small ulimit only for easy to trigger the error, but if Go packages with enough source files it will be a problem. And in special environment , concurrent open the file will lead to error: input/output error(EIO) , it lead to can not build code in such environment ,so I hope I can control the max open files by environment variable GOMAXPROCS.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

Closing in favor of #21621 since that one has a test case.

@golang golang locked and limited conversation to collaborators Aug 29, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.