New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bound goroutine counts #31

Merged
merged 1 commit into from Sep 11, 2018

Conversation

Projects
None yet
2 participants
@dbaggerman
Copy link
Collaborator

dbaggerman commented Sep 11, 2018

scc was starting to be capable of launching goroutines at a rate which
can lead to launching too many OS threads:

* linux-4.19-rc1 on a 16 core c5.4xlarge:
runtime: failed to create new OS thread (have 4090 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

this change bounds the number of goroutine workers to avoid this, while
still creating enough to saturate the CPU. This includes a marginal
improvement in performance, and appears to make the runtime much more
consistent:

* linux-4.19-rc1 on a 4 core c5.xlarge:
before: Time (mean ± σ):      4.680 s ±  0.727 s    [User: 17.920 s, System: 0.632 s]
after:  Time (mean ± σ):      4.532 s ±  0.005 s    [User: 17.340 s, System: 0.705 s]
bound goroutine counts
scc was starting to be capable of launching goroutines at a rate which
can lead to launching too many OS threads:

* linux-4.19-rc1 on a 16 core c5.4xlarge:
runtime: failed to create new OS thread (have 4090 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

this change bounds the number of goroutine workers to avoid this, while
still creating enough to saturate the CPU. This includes a marginal
improvement in performance, and appears to make the runtime much more
consistent:

* linux-4.19-rc1 on a 4 core c5.xlarge:
before: Time (mean ± σ):      4.680 s ±  0.727 s    [User: 17.920 s, System: 0.632 s]
after:  Time (mean ± σ):      4.532 s ±  0.005 s    [User: 17.340 s, System: 0.705 s]

@boyter boyter merged commit 859763f into boyter:master Sep 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment