runtime: failed to create new OS thread

[achawki]

Hi,
we see

fatal error: newosproc
runtime: failed to create new OS thread (have 5 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)

for various apps on our environment.
I guess it is related to #130522463
On one exemplary diego cell there are still enough resources:

# ps aux | wc -l 
511 
# ps -efT | wc -l 
2387 
# cat /proc/sys/kernel/threads-max 
128179
# cat /proc/<gdn pid>/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             64089                64089                processes
Max open files            65536                65536                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       64089                64089                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

We have seen this error also for very basic hello-world application. This error seems to appear when an app gets pushed (after the app container gets created).

Logging and/or test output

I couldn't find any suspicious garden log entries when this error message occurs.

Steps to reproduce

I couldn't reproduce the error message.

• cf v252
• garden-runc 1.1.1
• diego v1.6.2
• Linux kernel 4.4.0-53-generic
• bosh-aws-xen-hvm-ubuntu-trusty-go_agent 3312.12

[garden-gnome]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #140943849 runtime: failed to create new OS thread

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[achawki]

Could be also related #130206829

[georgethebeatle]

Hi there,

We have bumped up process limits as part of this story #137721673 . It is available in the v1.2.0 release. Could you please update to latest version of garden-runc and see if you are still hitting that problem?

[achawki]

Ok we will try it.

[julz]

Hi @achawki - I'm going to close this for now, do feel free to re-open if this re-occurs on 1.2.0+. Thanks!

[achawki]

@julz We still see the error message with diego v1.8.1 and garden-runc 1.2.0. The error seems not to have an effect on apps. It occurred also for very basic hello-world application.

I am not able to re-open this issue.

[julz]

Oops sorry @achawki I missed this, I'm reopening the issue

[holgero]

Hi, are there any news about this issue? I didn't see any changes in the #140943849 either, but I may be confused in which pivotal issue this is handled...

[jochenehret]

Hi, is there any progress on this issue? Or should we track another ticket instead?

Thanks and Best Regards,

Jochen.

[achawki]

We are meanwhile on cf v278/diego release v1.28.0 and we can't find the error message anymore.

[teddyking]

Just for future reference there's a whole load of context about how we eventually fixed this over here: https://www.pivotaltracker.com/n/projects/1003146/stories/148020689.

TL;DR either set runtime.GOMAXPROCS(1) or use a golang version > 1.9.