NUTCH-3058 Fetcher: counter for hung threads #820
Conversation
sebastian-nagel
commented
Jun 5, 2024
sebastian-nagel
commented
- count the number of hung threads in a fetcher job
- log and count the number of fetch items still queued when the "hard" timeout is reached
| for (int i = 0; i < fetcherThreads.size(); i++) { | ||
| FetcherThread thread = fetcherThreads.get(i); | ||
| if (thread.isAlive()) { | ||
| LOG.warn("Thread #{} hung while processing {}", i, |
There was a problem hiding this comment.
I wonder if we could improve this logging by explicitly stating that DEBUG-level logging can be turned on for Stack-level granularity. Otherwise the Nutch admin would never know that.
wdyt @sebastian-nagel ?
There was a problem hiding this comment.
by explicitly stating that DEBUG-level logging can be turned on for Stack-level granularity.
You mean something like: LOG.warn("(Turn on DEBUG logging to log stack traces of hung threads.)"); ?
Not a bad idea. But maybe we could print the stack traces with level WARN? Hung threads are a severe event, if not an issue. It means that for minimum 5 minutes there was no page fetched (with the default value of mapreduce.task.timeout). So, it seems ok, to log the stacks of hung threads on level WARN. I'll update the PR...
There was a problem hiding this comment.
Yes I think WARN logging here is called for. That sounds like a better solution. Thanks.
|
Once this has had more thorough peer review it would be great to add it to the Nutch Metrics documentation. I am +1 for the patch BTW. |
- log stack traces of hung threads with level WARN (instead of DEBUG)
|
Thanks for the discussion, @lewismc! The new counters were added to https://cwiki.apache.org/confluence/display/NUTCH/Metrics |
