Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: too many open files #21621

Closed
caglar10ur opened this issue Aug 25, 2017 · 20 comments

Comments

Projects
None yet
9 participants
@caglar10ur
Copy link
Contributor

commented Aug 25, 2017

What version of Go are you using (go version)?

1.9

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build185756508=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

  • clone github.com/caglar10ur/vic
  • since the original repository is under vmware org., rename the parent directory as vmware (mv $GOPATH/src/github.com/caglar10ur $GOPATH/src/github.com/vmware)
  • switch to go19 branch
  • run sudo -E make docker-engine-api

Please note the make target install swagger binary on your $GOPATH/bin so if you don't want that please use a vm/container or a temporary GOPATH

What did you expect to see?

No errors

What did you see instead?

Build fails with "too many open files", system uses default limits provided by ubuntu 16.04.

Our build step contains huge number of autogenerated files. Example; https://10ur.org/golang19.txt

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] sudo -E make docker-engine-api
Generating dependency set for cmd/imagec/
Generating dependency set for cmd/vic-machine/
Generating dependency set for cmd/vicadmin/
Generating dependency set for cmd/port-layer-server/
Generating dependency set for cmd/vic-dns/
Generating dependency set for cmd/tether/
Generating dependency set for cmd/vic-ui/
Generating dependency set for cmd/docker/
Generating dependency set for cmd/vic-init/
Generating dependency set for cmd/rpctool/
Generating dependency set for cmd/gandalf/
regenerating swagger models and operations for Portlayer API client...
done regenerating swagger models and operations for Portlayer API client...
regenerating swagger models and operations for Admiral API client...
done regenerating swagger models and operations for Admiral API client...
Building docker-engine-api server...
# github.com/vmware/vic/lib/config/dynamic/admiral/client/operations
lib/config/dynamic/admiral/client/operations/patch_resources_container_control_loop_id_parameters.go:0:0: open lib/config/dynamic/admiral/client/operations/patch_resources_container_control_loop_id_parameters.go: too many open files
Makefile:316: recipe for target 'bin/docker-engine-server' failed
make: *** [bin/docker-engine-server] Error 2

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] ulimit -n
1024

Bumping the number of open files limit to 2048 makes error go away.

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] ulimit -n 2048

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] ulimit -n
2048
[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] sudo -E make docker-engine-api
Generating dependency set for cmd/imagec/
Generating dependency set for cmd/vic-machine/
Generating dependency set for cmd/vicadmin/
Generating dependency set for cmd/port-layer-server/
Generating dependency set for cmd/vic-dns/
Generating dependency set for cmd/tether/
Generating dependency set for cmd/vic-ui/
Generating dependency set for cmd/docker/
Generating dependency set for cmd/vic-init/
Generating dependency set for cmd/rpctool/
Generating dependency set for cmd/gandalf/
regenerating swagger models and operations for Portlayer API client...
done regenerating swagger models and operations for Portlayer API client...
regenerating swagger models and operations for Admiral API client...
done regenerating swagger models and operations for Admiral API client...
Building docker-engine-api server...
[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)]

It works fine with go1.8

@ALTree

This comment has been minimized.

Copy link
Member

commented Aug 25, 2017

Dup of #21550 ?

@ianlancetaylor ianlancetaylor changed the title [go19] too many open files cmd/compile: too many open files Aug 25, 2017

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 25, 2017

Can you run the relevant go build or go install command with the -x option and show us the output? I'd like to see precisely which tool is generating the error. Thanks.

@ianlancetaylor ianlancetaylor added this to the Go1.9.1 milestone Aug 25, 2017

@caglar10ur

This comment has been minimized.

Copy link
Contributor Author

commented Aug 25, 2017

Sure, @ianlancetaylor please see below

...
mkdir -p $WORK/github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute/_obj/
cd /opt/go/src/github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute.a -trimpath $WORK -goversion go1.9 -p github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute -complete -buildid ec1fb3ca006c02f81994384699d724ddb8479fdc -importmap github.com/go-openapi/errors=github.com/vmware/vic/vendor/github.com/go-openapi/errors -importmap github.com/go-openapi
/runtime=github.com/vmware/vic/vendor/github.com/go-openapi/runtime -importmap github.com/go-openapi/runtime/client=github.com/vmware/vic/vendor/github.com/go-openapi/runtime/client -importmap github.com/go-openapi/strfmt=github.com/vmware/vic/vendor/github.com/go-openapi/strfmt -importmap golang.org/x/net/context=github.com/vmware/vic/vendor/golang.org/x/net/context -D _/opt/go/src/github.com/vmware/vic/lib/config/dynamic/a
dmiral/client/resources_compute -I $WORK -I /opt/go/pkg/linux_amd64 -pack ./get_resources_compute_id_parameters.go ./get_resources_compute_id_responses.go ./get_resources_compute_parameters.go ./get_resources_compute_responses.go ./patch_resources_compute_id_parameters.go ./patch_resources_compute_id_responses.go ./post_resources_compute_id_parameters.go ./post_resources_compute_id_responses.go ./post_resources_compute_param
eters.go ./post_resources_compute_responses.go ./put_resources_compute_id_parameters.go ./put_resources_compute_id_responses.go ./resources_compute_client.go
# github.com/vmware/vic/lib/config/dynamic/admiral/client/operations
lib/config/dynamic/admiral/client/operations/patch_config_event_topic_id_parameters.go:0:0: open lib/config/dynamic/admiral/client/operations/patch_config_event_topic_id_parameters.go: too many open files
...

The full log can be found at https://10ur.org/build.log

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2017

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2017

@caglar10ur

This comment has been minimized.

Copy link
Contributor Author

commented Aug 27, 2017

@davecheney ah yeah, the original repo is under vmware not under caglar10ur :/ An ugly workaround like following should do trick;

mv /home/dfc/src/github.com/caglar10ur/ /home/dfc/src/github.com/vmware/

Edit: Updated the original report to reflect this.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2017

@caglar10ur

This comment has been minimized.

Copy link
Contributor Author

commented Aug 28, 2017

Unfortunately go build -x -i ./cmd/docker will work only if you run swagger to generate those files before, that's why I suggested using make target as it does that for you.

One thing that I realized just now is, the make target also tries to install swagger on your GOPATH so if you don't want that then I suggest using a vm/container or a temporary GOPATH

@caglar10ur

This comment has been minimized.

Copy link
Contributor Author

commented Aug 28, 2017

@davecheney the same set of files are also visible on the build.log that I provided (#21621 (comment)) and unless I made a mistake to extract them via grep, I believe we are talking about 1834 files.

[conur@vir:~/Documents] curl https://10ur.org/build.log 2> /dev/null | grep put_resources_sub_network | grep pack | awk -F"-pack" '{print $2}' | gsed -e 's/\s\+/\n/g' | wc -l
    1834
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2017

@dcheney-atlassian

This comment has been minimized.

Copy link

commented Aug 28, 2017

Here is a reproduction which fails reliably with go 1.9 and appears to pass reliably in go 1.8
bug.tar.gz

deadwood(~/src/github.com/davecheney/bug) % go build .
# github.com/davecheney/bug
./alec.go:0:0: open ./alec.go: too many open files
deadwood(~/src/github.com/davecheney/bug) % go build .
# github.com/davecheney/bug
./apology.go:0:0: open ./apology.go: too many open files
deadwood(~/src/github.com/davecheney/bug) % go build .
# github.com/davecheney/bug
./adjourning.go:0:0: open ./adjourning.go: too many open files
deadwood(~/src/github.com/davecheney/bug) % go1.8  build .
deadwood(~/src/github.com/davecheney/bug) % go1.8  build .
deadwood(~/src/github.com/davecheney/bug) % go1.8 build .
deadwood(~/src/github.com/davecheney/bug) % 
@valyala

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

The odd thing is I couldn't trigger the error consistently. I could trigger
it enough to convince myself that there is a regression, but I'm wondering
why sometimes the build works, and other times it fails for that package.
It would appear the issue is more complicated than just opening all the
files passed to the compiler at once, then processing them.

This is a simple race between concurrently running goroutines that parse distinct files. Sometimes they open too many files (I suppose the probability increases on cold file cache), sometimes they manage to parse and close old files before other goroutines start opening new files.

Try the patch https://go-review.googlesource.com/c/go/+/57751 that limits the number of concurrently open files during package parsing to GOMAXPROCS.

@mvdan

This comment has been minimized.

Copy link
Member

commented Aug 29, 2017

Should we perhaps close this in favor of #21550, or close that one in favor of the discussion here? Seems to me like they're both about the same underlying issue.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2017

I think https://golang.org/cl/57751 is the kind of CL we should try to get into the 1.9.1 release.

@davecheney You have a -2 on that CL pending discussion; are you OK with dropping that?

@gopherbot

This comment has been minimized.

Copy link

commented Aug 31, 2017

Change https://golang.org/cl/57751 mentions this issue: cmd/compile: Limit the number of simultaneously opened files to avoid EMFILE/ENFILE errors

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Sep 14, 2017

Reopen for merge to 1.9.1.

@gopherbot

This comment has been minimized.

Copy link

commented Sep 14, 2017

Change https://golang.org/cl/63752 mentions this issue: [release-branch.go1.9] cmd/compile: limit the number of simultaneously opened files to avoid EMFILE/ENFILE errors

@rsc rsc modified the milestones: Go1.9.1, Go1.9.2 Oct 4, 2017

@rsc rsc reopened this Oct 13, 2017

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 13, 2017

If the hope is that CL 63752 is going to fix #21713 as well, the sem <- struct{}{} should be before the goroutine creation, not inside the new goroutine. Otherwise looks good for Go 1.9.2.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 13, 2017

CL 63752 OK for Go 1.9.2. (cherry-pick, want to record original if it applies cleanly)
CL 57751 OK for Go 1.9.2.

gopherbot pushed a commit that referenced this issue Oct 25, 2017

[release-branch.go1.9] cmd/compile: limit the number of simultaneousl…
…y opened files to avoid EMFILE/ENFILE errors

If the Go packages with enough source files,it will cause EMFILE/ENFILE error,
Fix this by limiting the number of simultaneously opened files.

Fixes #21621

Change-Id: I8555d79242d2f90771e37e073b7540fc7194a64a
Reviewed-on: https://go-review.googlesource.com/57751
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-on: https://go-review.googlesource.com/63752
Run-TryBot: Russ Cox <rsc@golang.org>
@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2017

go1.9.2 has been packaged and includes:

The release is posted at golang.org/dl.

— golang.org/x/build/cmd/releasebot, Oct 26 21:09:09 UTC

@rsc rsc closed this Oct 26, 2017

@golang golang locked and limited conversation to collaborators Oct 26, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.