Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "go install -a" needs quite a few user processes and doesn't cope if they are restricted #16713

Closed
garyhouston opened this issue Aug 16, 2016 · 6 comments

Comments

Projects
None yet
5 participants
@garyhouston
Copy link

commented Aug 16, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    1.7
  2. What operating system and processor architecture are you using (go env)?
    amd64, linux
  3. What did you do?
    Rebuilding a program that was fine in 1.6, with:
    go install -a
  4. What did you expect to see?
    normal compilation
  5. What did you see instead?
    failed with:
# sort
runtime: failed to create new OS thread (have 3 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

(followed by stack dump and several similar messages.)

environment:

ulimit -u gives 30 (it's not my server and I can't increase it)

setting GOMAXPROCS=1 was a work-around that gave successful compilation.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 16, 2016

I think you'll probably have to stick with your workaround. 30 is quite low. It's conceivable you have more processors than that, in which case Go's default GOMAXPROCS would already violate your ulimit.

@ianlancetaylor? @aclements?

@bradfitz bradfitz changed the title build with 1.7 using too many processes runtime: 1.7 hits max user processes earlier than 1.6 Aug 16, 2016

@garyhouston

This comment has been minimized.

Copy link
Author

commented Aug 16, 2016

4 processors. Is there a limit to the number of user processes the build system is likely to need, or does it just increase with the number of files in the project?

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2016

-p=1 will limit the number of concurrent compile jobs
GOMAXPROCS=1 should limit the number of concurrently running goroutines
(including gc workers).

This should reduce the number of threads on, say a 4 processor system, by a
factor of 4^2

On Tue, Aug 16, 2016 at 3:29 PM, Gary Houston notifications@github.com
wrote:

4 processors. Is there a limit to the number of user processes the build
system is likely to need, or does it just increase with the number of files
in the project?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#16713 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA9Z__odVtgLOA0pJbzkRppEsVN1Xks5qgUqkgaJpZM4Jk8Hh
.

@aclements

This comment has been minimized.

Copy link
Member

commented Aug 22, 2016

@garyhouston, you say that it hits the limit "earlier" than in 1.6. I'm not at all surprised that it hits a limit of 30, but I'm somewhat surprised that it changed from 1.6 to 1.7. At what point did it fail in 1.6?

It could simply be that 1.7 has a slightly different package graph, which causes it to hit a more concurrent part earlier.

At any rate, I think your workaround (or using -p) is the right answer to an extremely low process limit.

@garyhouston

This comment has been minimized.

Copy link
Author

commented Aug 22, 2016

Thanks, I made an unwarranted assumption that it "was fine in 1.6", because I never encountered a problem like this previously. But perhaps I didn't use go install -a in 1.6, or maybe my code has changed in some significant way.

I reinstalled 1.6.3 and tried the same command, and got:
runtime/cgo: pthread_create failed: Resource temporarily unavailable

Today I get the same error in 1.7, instead of the original error that helpfully pointed out the ulimit issue. A regular install, without -a, after modifying only a few files, doesn't have any problem.

@garyhouston garyhouston changed the title runtime: 1.7 hits max user processes earlier than 1.6 runtime: "go install -a" needs quite a few user processes and doesn't cope if they are restricted Aug 22, 2016

@aclements

This comment has been minimized.

Copy link
Member

commented Aug 22, 2016

30 threads is not remotely "quite a few" by today's standards. It's not really reasonable for Go to attempt to automatically deal with such a constrained environment. Given that there appear to be two viable workarounds and this is not a regression from early Go versions, I'm going to go ahead and close this issue.

@aclements aclements closed this Aug 22, 2016

@golang golang locked and limited conversation to collaborators Aug 22, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.