9.0.1
·
15 commits
to main
since this release
a11b0c6
This commit was signed with the committer’s verified signature.
freekmurze
Freek Van der Herten
What's fixed
When the crawler reached its time limit, crawl limit, or was interrupted via a signal, any HTTP requests that were still in-flight in the Guzzle pool would eventually fail (e.g. due to their per-request timeout) and get reported to crawlFailed. These URLs were dispatched but never truly crawled, so reporting them as failures was misleading.
The CrawlRequestFailed handler now checks whether the crawler has reached its limits before reporting a failure. If it has, the failure is silently ignored.
Regular request failures (timeouts, connection errors, server errors) that happen during normal crawling are still reported as before.