Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/cmd/godoc: ignored "too many open files" errors can cause incomplete /pkg/ subdirectories listings #14252

Closed
dmitshur opened this issue Feb 8, 2016 · 6 comments

Comments

Projects
None yet
7 participants
@dmitshur
Copy link
Member

commented Feb 8, 2016

What version of Go are you using (go version)?

go version go1.6rc2 darwin/amd64

What operating system and processor architecture are you using?

OS X 10.11.3 (64-bit).

What did you do?

  1. Have a GOPATH workspace with many, many packages.
  2. Run godoc -http=localhost:6060.
  3. Visit http://localhost:6060/pkg/crypto/ (it's used as an example, it affects all packages equally).

What did you expect to see?

A complete list of subdirectories, with their package synopses.

image

What did you see instead?

Randomly incomplete listing of subdirectories, and some missing package synopses.

image

image

@dmitshur

This comment has been minimized.

Copy link
Member Author

commented Feb 8, 2016

I've looked into why it happens. At first I thought it might be some race condition, but I've learned it's not.

Instead, the problem is here:

https://github.com/golang/tools/blob/fe74a4186116b8d7dd38a723993e0d84f8834b34/godoc/dirtrees.go#L107-L127

Because all GOROOT and GOPATH workspaces are walked in parallel (likely with some concurrency limits), it may run into "too many files open" limits and erroring out. The linked code segment ignores all situations where parsing the .go files results in errors (there's no if err != nil { ... } section).

I've added a simple else if err != nil { log.Println(err) } section there and I saw many "too many files open" errors occur both on .go files in GOROOT as well as in GOPATH during my test.

@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Feb 8, 2016

@rsc rsc modified the milestones: Go1.8, Go1.7 May 18, 2016

@quentinmit quentinmit added the NeedsFix label Oct 10, 2016

@rsc rsc modified the milestones: Go1.9, Go1.8 Nov 11, 2016

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 6, 2017

@adg improved this in https://go-review.googlesource.com/24060

I'll send a bit more.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 6, 2017

CL https://golang.org/cl/47651 mentions this issue.

@bradfitz bradfitz added the help wanted label Jul 6, 2017

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 Jul 6, 2017

gopherbot pushed a commit to golang/tools that referenced this issue Jul 6, 2017

godoc: include dir listing in rate limit, log dir reading errors
Updates golang/go#14252

Change-Id: I3ce2a410708900f5c80d15e722b410855fd9c571
Reviewed-on: https://go-review.googlesource.com/47651
Reviewed-by: Dmitri Shuralyov <shurcool@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2017

I think this has been addressed. At the very least, I have a pretty large source tree (every Github checkout I do in any language is in GOPATH) and I've never seen this issue, nor a godoc server with randomly missing files.

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2017

Closing, feel free to reopen if this is still a problem, or there's a way to reproduce on tip.

@kevinburke kevinburke closed this Oct 2, 2017

@dmitshur

This comment has been minimized.

Copy link
Member Author

commented Oct 2, 2017

I suspect that the 2 commits to help this issue (golang/tools@a2a5522 and golang/tools@72ed06f) were sufficient to resolve it. If I ever run into this again, I'll post an update.

@golang golang locked and limited conversation to collaborators Oct 2, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.