-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
explicitly call lifecycle stop #1627
Conversation
@drcrallen , as you are most familiar with this, can you also double check this change ? |
👍 |
Here is the task log https://gist.github.com/guobingkun/6025d9b0afa3edfe8e6e for the realtime task. At 2015-08-13T22:34:22,824, TimedShutoffFirehose was shutting down. |
@guobingkun Thanks for the log. @nishantmonu51 Can this add a test in ITRealtimeIndexTaskTest to look for this error case? |
Before merging this, I would much rather get a thread dump of the offending tasks and see if they can be unstuck. |
👍 |
+1 |
also, @guobingkun is going to try this out on our test cluster to see if it fixes the problem we noticed |
Originally introduced in #1524 |
found realtime tasks getting stuck on shutdown even after status being shown as SUCCESS.
On further investigation found that the vm was waiting for all the daemon threads to shutdown.
was able to reproduce this locally, calling lifecycle.shutdown explicitly and not relying on the shutdown hook fixes the issue.