HTTP transport, CLI, and redirects as host rules - #1
Merged
Conversation
A 3xx path used to be followed in place: the destination's full HTML was
written under the OLD path (duplicate content, no canonical) and the
destination itself was never rendered at its own URL unless linked.
Now a redirect is a record ({ from, to, status }, one per hop) and its
same-origin target enters the queue as a page in its own right. The
redirected path gets a meta-refresh stub pointing at the chain's final
destination. A redirect to another spelling of the same page
(/posts -> /posts/) is followed in place, not recorded. Cycles terminate
because each path is crawled once; maxRedirects has nothing left to bound
and is removed.
redirects() integration emits a _redirects rules file (Netlify /
Cloudflare Pages) and declares handlesRedirects, which stops the engine
writing stubs: Cloudflare applies rules regardless of files, but Netlify
lets an existing file shadow an unforced rule. redirectStubs controls
stubs directly. PrerenderContext exposes live pages and redirects views.
Co-authored-by: Cursor <cursoragent@cursor.com>
httpTransport(target) re-addresses the crawl's requests to a running server and hands redirects back unfollowed; moduleTransport(entry) / loadHandler(entry) import a handleRequest / fetch / default.fetch module (the Vite plugin now uses it). The engine's Transport contract was always the agnostic pitch; these make it true outside Vite. prerender-crawler <target> --out <dir>: a CLI over both transports with seeds, mode, concurrency, interval, retries, link following, redirect handling and --continue. For an HTTP target the crawl origin is the target's so absolute links count as same-origin. Co-authored-by: Cursor <cursoragent@cursor.com>
interval spaced the slots workers claimed on a shared timeline. A start whose timer fired late (busy event loop — CI, or a heavy render) could be followed by the next worker's on-time slot less than interval later. Now a worker also holds until interval has passed since the last ACTUAL start, and the timeline moves to that start. Monotonic clock throughout; the test that measured with performance.now against a Date.now pacer no longer disagrees with itself by a millisecond. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two of the gaps between what the README claims and what was usable outside Vite + Solid.
Transports and CLI
httpTransport(target, { headers?, fetch? })— prerender a running server (preview server, container, staging deploy). Requests are re-addressed to the target's origin; redirects come back as the server's 3xx so the engine handles them.moduleTransport(entry)/loadHandler(entry)— import ahandleRequest/fetch/default.fetchmodule. The Vite plugin now uses this instead of its private copy.npx prerender-crawler <target> --out <dir>over both, with seeds, mode, concurrency, interval, retries,--no-links,--redirects,--continue,--flat.Redirects
Previously an internal redirect wrote the destination's full HTML under the old path and never rendered the destination at its own URL unless linked — duplicate content with no canonical, and a correctness gap on static hosts (the crawl followed the redirect; production had no rule for it).
Now a 3xx path is a
RedirectRecord(one per hop) and its target is crawled as a page of its own; the old path gets a meta-refresh stub pointing at the chain's end. Same-page respellings (/posts → /posts/, what static servers do) are followed in place.redirects()emits a_redirectsfile and declareshandlesRedirects, which suppresses stubs — Cloudflare applies rules regardless of files but Netlify lets a file shadow an unforced rule (and rejects Cloudflare-incompatible!), so no stub is the one configuration that works on both.maxRedirectsis removed (nothing left to bound).PrerenderContextgains livepages/redirects. Minor bump.Verified beyond the unit tests: the built
binagainst the example'sdist/server/server.js(in-process) and against a static file server over HTTP (trailing-slash redirects followed in place, nothing spuriously recorded).Made with Cursor