libgomp: Thread creation failed: Resource temporarily unavailable

[markstur]

Describe the bug

Under high concurrency testing, latency got slower, timeouts started to happen, and we get intermittent " libgomp: Thread creation failed: Resource temporarily unavailable".

Not sure how best to characterize the "concurrency", but on CPU I'd say we test w/ 100 concurrent requests and see latency degrade and at 500 we start getting errors.

This is a regression that showed up in 0.25.6.

NOTE! It looks like configuring server_thread_pool_size: 5 fixes it. In 0.25.6 that default was changed from 5 to 100 (related to HTTP issues). We are only using gRPC.

Platform

Please provide details about the environment you are using, including the following:

• Interpreter version:
• Library version: Comparing 0.25.x versions

Sample Code

Please include a minimal sample of the code that will (if possible) reproduce the bug in isolation

Expected behavior

In my case, it seems like the default should be the better performing 5.

Ideally limits would prevent the resource error from ever happening (but don't know how possible that is to fix).

Observed behavior

What you see happening (error messages, stack traces, etc...)

Additional context

Add any other context about the problem here.