RFC: Automatic pagination detection for crawls (reduce manual “next page” rules)

[vigneshwarrvenkat]

Feature description

Detect common pagination patterns (e.g. rel="next", “next”/localized link text, numbered pages, query params like ?page=) and optionally emit follow-up requests with configurable confidence / overrides.

Motivation

Most crawls repeat the same boilerplate; mistakes here waste bandwidth and time.

Proposal (high level)

• Opt-in API at spider or LinkExtractor level; verbose logging when a pattern is chosen.
• Hook naturally into CrawlSpider / LinkExtractor.

Related

• ROADMAP.md: “Add functionality to automatically detect pagination URLs”

Open questions

• How to handle JS-only pagination (document as unsupported vs hook for manual session)?
• False-positive policy: conservative defaults vs “aggressive mode”?

(I searched existing feature requests for this topic.)