Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception in ResponseThreadRunnable causes REST API to fail #11722

Closed
paykin opened this issue Nov 2, 2017 · 3 comments
Closed

Exception in ResponseThreadRunnable causes REST API to fail #11722

paykin opened this issue Nov 2, 2017 · 3 comments

Comments

@paykin
Copy link

@paykin paykin commented Nov 2, 2017

When there is an exception while processing single command, entire thread got killed. It causes unrecoverable condition when system does not process any incoming REST command and causing memory leak (incoming commands are accumulated in blockingQueue).
The only resolution is restart of entire system.

In our case heavy system swapping caused this situation to occur. Here is the full stack trace (for version 3.6.2):

Exception in thread "hz.MY_CLUSTER.ascii.service.response" java.lang.NullPointerException
at com.hazelcast.internal.ascii.TextCommandServiceImpl$ResponseThreadRunnable.run(TextCommandServiceImpl.java:379)
at java.lang.Thread.run(Thread.java:745)

Here is the PR that fixes it: #11721

@paykin paykin changed the title Exception ResponseThreadRunnable causes REST API to fail Exception in ResponseThreadRunnable causes REST API to fail Nov 2, 2017
@mmedenjak mmedenjak added this to the 3.10 milestone Nov 10, 2017
@taburet taburet added Team: Core and removed Team: Core labels Jan 22, 2018
@jerrinot jerrinot self-assigned this Feb 28, 2018
@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Feb 28, 2018

actually I am not sure if just catching/logging the exception is the right thing to do. in this case it would just a hide an issue with a write handler initialization.

edit: from the other hand - catching errors in a "command loop" is consistent with what the operation running is doing.

@paykin
Copy link
Author

@paykin paykin commented Feb 28, 2018

@jerrinot you may want to change behavior during initialization, but during runtime you do not want that single exception will make entire system non-functional without any option to recover (full restart does not count).

@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Mar 5, 2018

fixed by #12481

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants