Third part of #867, after the Retry-After handling (#784, done in #1973) and the adaptive back-off for headerless rate limits (#1106).
Today a robots.txt Crawl-delay above fetcher.max.crawl.delay has no good outcome: by default the URL is marked Status.ERROR (error.cause=crawl_delay) and the host is never crawled, with fetcher.max.crawl.delay.force it is fetched at a pace that violates what robots asked. Short delays are fine where they are, inside the fetcher queues; it's the long tail that loses either way. And these sites are exactly the ones that mean it - crawl-delay is the most-complied-with robots directive in the wild (https://arxiv.org/abs/2505.21733).
The frontier can do better. With force enabled the fetcher would report the requested delay in the status metadata, and the queue-stream consumer (QueueRegulatorBolt) would pass it to URLFrontier via setDelay: the host becomes crawlable at the pace robots asked for, and the compliance that force alone breaks is restored at the source. Same split as Nutch-style crawlers moved to long ago: politeness state lives with the queue owner, not the workers.
Two things worth calling out. Unlike blockQueueUntil, setDelay does not expire on its own, and setDelay(key, 0) erases a server-side default for that queue - so when a site later removes its Crawl-delay, the old pace simply stays (throughput lost on that host, never impoliteness) rather than guessing at a reset. And the value needs a cap of its own: SC disables the parser-side limit on crawl-delays, and sites do ask for absurd numbers.
Third part of #867, after the Retry-After handling (#784, done in #1973) and the adaptive back-off for headerless rate limits (#1106).
Today a robots.txt
Crawl-delayabovefetcher.max.crawl.delayhas no good outcome: by default the URL is markedStatus.ERROR(error.cause=crawl_delay) and the host is never crawled, withfetcher.max.crawl.delay.forceit is fetched at a pace that violates what robots asked. Short delays are fine where they are, inside the fetcher queues; it's the long tail that loses either way. And these sites are exactly the ones that mean it - crawl-delay is the most-complied-with robots directive in the wild (https://arxiv.org/abs/2505.21733).The frontier can do better. With
forceenabled the fetcher would report the requested delay in the status metadata, and the queue-stream consumer (QueueRegulatorBolt) would pass it to URLFrontier via setDelay: the host becomes crawlable at the pace robots asked for, and the compliance thatforcealone breaks is restored at the source. Same split as Nutch-style crawlers moved to long ago: politeness state lives with the queue owner, not the workers.Two things worth calling out. Unlike
blockQueueUntil,setDelaydoes not expire on its own, andsetDelay(key, 0)erases a server-side default for that queue - so when a site later removes itsCrawl-delay, the old pace simply stays (throughput lost on that host, never impoliteness) rather than guessing at a reset. And the value needs a cap of its own: SC disables the parser-side limit on crawl-delays, and sites do ask for absurd numbers.