fix(core): keep crawler enqueue limits when options carry explicit undefined - #3927
Merged
Conversation
…defined
`BasicCrawler`'s context-bound `enqueueLinks` built its call as `{ requestQueue,
robotsTxtFile, respectRobotsTxtFile, onSkippedRequest, limit, ...options }`, so a
user options object carrying an explicitly present `undefined` key silently
discarded the computed value:
await enqueueLinks({ urls, limit: config.limit }); // config.limit is not set
Neither `ow.optional.number` nor TypeScript (without `exactOptionalPropertyTypes`)
rejects an explicit `undefined`, so the `calculateEnqueuedRequestLimit()` result
was dropped - the queue kept growing past the remaining `maxRequestsPerCrawl`
budget and the `enqueueLimit` skips were never reported to `onSkippedRequest`.
`maxRequestsPerCrawl` is still enforced as a stop condition, so the effect was
queue bloat and missing reporting rather than an unbounded crawl. The same
applied to `robotsTxtFile` and `respectRobotsTxtFile` (robots.txt filtering
silently disabled at enqueue time), and to `requestQueue`, where the explicit
`undefined` made `ow` throw instead.
The crawler-derived values now merge after the user options instead of before:
- `limit` becomes `min(user limit, remaining maxRequestsPerCrawl budget)`, which
is what `addRequests()` and the per-crawler `*CrawlerEnqueueLinks` helpers have
always done - this path was the outlier
- `onSkippedRequest` is composed, so the crawler reporting keeps running when the
user passes their own callback, instead of being replaced by it
- `requestQueue`, `robotsTxtFile` and `respectRobotsTxtFile` fall back with `??`,
so a defined user value still wins
The enqueue-limit log message now also distinguishes the two causes, instead of
blaming the user's `limit` when the `maxRequestsPerCrawl` clamp is what bit.
barjin
approved these changes
Aug 4, 2026
…contexts CheerioCrawler and BrowserCrawler pre-clamped `options.limit` before calling the bound `enqueueLinks`, so the enqueueLimit log attributed skips to an explicit enqueueLinks limit even when only the maxRequestsPerCrawl budget was clamping. The bound path clamps authoritatively, so the pre-clamp is dropped there (jsdom/linkedom keep theirs, they use the non-bound path). Also documents that the crawler-level and per-call onSkippedRequest callbacks compose instead of replacing each other.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #3924, same bug class one level down.
BasicCrawler's context-boundenqueueLinksbuilt its call as{ requestQueue, robotsTxtFile, respectRobotsTxtFile, onSkippedRequest, limit, ...options }, so a user options object carrying an explicitly presentundefinedkey silently discarded the computed value:Neither
ow.optional.numbernor TypeScript (withoutexactOptionalPropertyTypes) rejects an explicitundefined, so thecalculateEnqueuedRequestLimit()result was dropped — the queue kept growing past the remainingmaxRequestsPerCrawlbudget and theenqueueLimitskips were never reported toonSkippedRequest.maxRequestsPerCrawlis still enforced as a stop condition viaisMaxPagesExceeded(), so the effect was queue bloat and missing reporting rather than an unbounded crawl. The same applied torobotsTxtFileandrespectRobotsTxtFile(robots.txt filtering silently disabled at enqueue time), and torequestQueue, where the explicitundefinedmadeowthrow instead.Unlike #3924 this isn't a pure reordering, so the semantics for each computed key:
limit— nowmin(user limit, remaining maxRequestsPerCrawl budget). This is whataddRequests()(maxNewRequestsafter the spread) and the per-crawler*CrawlerEnqueueLinkshelpers have always done; this path was the outlier. A user limit larger than the remaining budget is now capped, which is a user-visible change.onSkippedRequest— composed instead of replaced: the crawler's reporting (logging, crawler-levelonSkippedRequest) runs, then the user's callback. Previously passing a callback toenqueueLinkssilently disabled the crawler-level one.requestQueue,robotsTxtFile,respectRobotsTxtFile—??fallbacks, so a defined user value still wins.The enqueue-limit log message now also distinguishes the two causes, instead of blaming the user's
limitwhen themaxRequestsPerCrawlclamp is what bit.Not covered here:
JSDOMCrawlerandLinkeDOMCrawlernever routecontext.enqueueLinksthrough this wrapper at all (they call the standaloneenqueueLinks()directly, unlikeCheerioCrawler/BrowserCrawlerwhich forward the bound context function), socrawlDepthinjection andmaxCrawlDepthare inert there. Separate issue, separate fix.