Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infinite loop in case of @pony_os_accept() error #3045

Open
slfritchie opened this issue Oct 29, 2019 · 0 comments
Open

Infinite loop in case of @pony_os_accept() error #3045

slfritchie opened this issue Oct 29, 2019 · 0 comments

Comments

@slfritchie
Copy link
Contributor

Today's master branch's commit 1ea9650 for https://github.com/WallarooLabs/wallaroo/blob/master/lib/wallaroo/core/data_channel/data_channel_listener.pony#L174-L187 shows that if @pony_os_accept() call fails, then there's an immediate retry. For some error conditions, this is fine, but for other conditions, such as EMFILE, this is bad.

In the case of EMFILE, the process has it its file descriptor limit. Odds are extremely high that EMFILE will happen again in this kind of tight loop. Also, the condition can last long enough that the actor's behavior call doesn't return promptly, which can occupy an entire Pony runtime scheduler thread ... and, worst case, occupy all available scheduler threads. Credit to @JONBRWN for discovering this scenario with a 30 worker Wallaroo cluster and a Linux ulimit -n value of 1024.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant