FetchDNS: Keep dnsjava selector thread out of ToePool #444
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since the cache is global we only need to configure it once, not every time we start a job. By moving it we keep the code that interfaces with dnsjava centralized in FetchDNS. Looks like dnsjava no longer uses a cleaner thread so remove the comment referring to it.
dnsjava will create its global selector thread on first DNS lookup. Since the first lookup occurs on a toe thread then this will cause the selector thead to join the job's toe thread group. This is a problem because when the job is terminated ToePool sends an interrupt to all threads in the thread group which the dnsjava selector thread is not expected and never clears. The interrupted state then causes select() to return immediately instead of sleeping so it consumes 100% CPU. So workaround this by doing a dummy lookup when FetchDNS is loaded, this ensures the first lookup happens on the web server thread that handles the job build action instead of a toe thread. Fixes: 045b251 ("Update to latest version of dnsjava")
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #425, the dnsjava selector thread using 100% CPU because it was being created in the toe thread pool and then interrupted when the job was terminated. By doing a dummy lookup in a static block we ensure the selector thread is launched from a normal thread rather than from a toe thread.
There was an existing workaround in CrawlController for a similar problem with the dnsjava cache cleaner thread. This cleaner thread no longer exists in the current version of dnsjava but I assume we still want to set the cache size anyway. I moved this code to FetchDNS as well to keep the code that interfaces with dnsjava centralized. Since the cache is global it only needs to be configured once, not on every job launch.