Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FetchDNS: Keep dnsjava selector thread out of ToePool #444

Merged
merged 2 commits into from
Nov 27, 2021
Merged

FetchDNS: Keep dnsjava selector thread out of ToePool #444

merged 2 commits into from
Nov 27, 2021

Commits on Nov 1, 2021

  1. FetchDNS: Move DNS cache init from CrawlController 

    Since the cache is global we only need to configure it once, not every
time we start a job. By moving it we keep the code that interfaces with
dnsjava centralized in FetchDNS.

Looks like dnsjava no longer uses a cleaner thread so remove the comment
referring to it.
    @ato
    ato committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    513c5fc View commit details
    Browse the repository at this point in the history

  2. FetchDNS: Keep dnsjava selector thread out of ToePool 

    dnsjava will create its global selector thread on first DNS lookup.
Since the first lookup occurs on a toe thread then this will cause the
selector thead to join the job's toe thread group. This is a problem
because when the job is terminated ToePool sends an interrupt to all
threads in the thread group which the dnsjava selector thread is
not expected and never clears. The interrupted state then causes
select() to return immediately instead of sleeping so it consumes
100% CPU.

So workaround this by doing a dummy lookup when FetchDNS is loaded,
this ensures the first lookup happens on the web server thread that
handles the job build action instead of a toe thread.

Fixes: 045b251 ("Update to latest version of dnsjava")
    @ato
    ato committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    1832c5e View commit details
    Browse the repository at this point in the history