Use daemon threads for connection backoff resetter #244
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently we've added a connection backoff resetter as a way to mitigate long gRPC backoff intervals (look here for details).
The way resetter works is that we have a thread pool that periodically resets connection backoff allowing threads to detect backends faster.
Initially those threads in the thread pool used default thread factory resulting in creation of user threads (as opposed to daemon threads) and were blocking JVM shutdown in case of failure in main program.
This caused unwanted behavior and displayed on our java canary as a "stuck jvm" with crashed main in case if the process was unable to startup properly (e.g. register the namespace).
With this change threads are made daemon threads and will no longer block jvm exit.
Example canary startup log when backend is unavailable after the change (expected behavior, JVM exits if it's unable to create a namespace):