Partial Prefetching: skip the embedded runtime prefetch (p) for blocking-metadata bot renders, and when it repeats the static shell

[trevorlasn]

Goals

1. Leave the embedded runtime prefetch stream (the RSC payload's p field) out of documents rendered with blocking metadata for a user agent that matches htmlLimitedBots. Those agents never hydrate, so the second render and its bytes are pure cost.
2. Avoid embedding the stream when the runtime prerender adds nothing beyond the static stage that the document already carries.
3. Let an app keep Partial Prefetching for links to a route while that route's documents skip the embed.

Non-Goals

1. Changing how links prefetch App Shells, or what <Link prefetch={true}> resolves.
2. Changing streamed metadata for browsers, or the htmlLimitedBots list.
3. Changing the embed for browsers on routes whose runtime prerender carries runtime content.

Background

With cacheComponents: true and partialPrefetching: true on Next.js 16.4.0, a production document render takes the cacheComponents && cachedNavigations branch in server/app-render/app-render.js. In Partial mode the branch waits for cacheSignal.cacheReady(), runs spawnRuntimePrefetchWithFilledCaches, and embeds the result as the payload's p field, so the client router can cache runtime-prefetchable content during hydration.

getPrefetchingModeForPage returns Partial as soon as renderOpts.partialPrefetching is set, before it reads any segment prefetch export. base-server.js sets renderOpts.partialPrefetching once, from the config. Nothing on this path reads the user agent; serveStreamingMetadata only decides where the metadata goes. A segment's prefetch export governs links to that segment. So an app cannot keep a document free of the embed without dropping Partial Prefetching for every link to the route. 16.5.0-canary.10 has the same gate.

Two cases where nobody reads the stream:

1. Blocking-metadata bots. A request that matches htmlLimitedBots is rendered with blocking metadata for an agent that does not run JavaScript. It never hydrates, so the second render and the embedded bytes are wasted. A serverless function returns the document uncompressed, so the bytes are paid in full between the function and the edge.
2. A stream that repeats the shell. In some routes, every runtime read stalls the runtime prerender at the page's shell boundary: here, data reads that call connection() below a Suspense boundary. The stream then repeats the root layout and the static shell that the document already carries. On our routes its largest rows are <html>, <body>, the footer, a search trigger, a toaster and the page skeletons. Its only page-specific content is the route parameter and one title.

Measured on 16.4.0 with next build and next start: one build, with partialPrefetching on, and off at next start. The documents were requested with the bot user agent meta-externalagent/1.1, and the bytes are the document as served. Row 0 carries "p":"$<id>", and the stream is the <id>:o<hex>, chunks, so the "Stream" column is the sum of those chunk lengths. CPU is the server process's CPU time per document, over 20 alternating request pairs.

Route	partialPrefetching: true, raw / gzip bytes	false, raw / gzip bytes	Stream, decoded bytes	Stream share of gzip	CPU per document, true / false
/event/[slug]	515,055 / 70,552	469,723 / 60,883	41,156	13.7%	52.5 / 41.5 ms
/profile/[username]	524,835 / 95,833	443,100 / 80,496	72,042	16.0%	74.5 / 60.0 ms
/sports	499,136 / 78,701	362,742 / 58,524	121,459	25.6%	39.0 / 32.5 ms
/skill-leaderboard	777,098 / 121,400	645,442 / 96,960	118,389	20.1%	44.0 / 43.0 ms

Browsers get the same stream: 15.5-25.5% of each of these documents gzipped. In production we serve about 28,000 crawler documents a day, each 0.5-1.0 MB uncompressed.

The alternatives available today:

• partialPrefetching: false. This applies app-wide, or per route by exporting prefetch = 'partial' from every other route. It removes the stream but regresses navigation for readers. On a page with 126 links to one dynamic route, prefetching went from 5 requests and 7,843 B to 8 requests and 25,432 B. A tap on an unprefetched link took 85-96 ms instead of 44-50 ms to commit the URL. The docs also say the option goes away in the next major.
• experimental.cachedNavigations: false. It skips the branch, but it also drops the hydration cache of the static stage and the staged navigation responses on every route.
• Calling connection() early for bot requests. It only moves where the runtime prerender stops. The layout and shell that the stream repeats sit above that point.

Proposal

1. In the document branch, decide the embed per request. When serveStreamingMetadata is false because the user agent matched htmlLimitedBots, do not create startRuntimePrefetchRender, and leave p out of that response. The render already knows it is serving an agent that will not hydrate.
2. Optionally, skip the embed when the runtime prerender produced nothing beyond the static stage the document already describes with l.
3. Optionally, add a segment or config switch that keeps Partial Prefetching for links but turns off the embedded runtime prefetch in that segment's documents.

A fix can be checked with the same reads: request a document with a bot user agent, and sum the chunk lengths of the row that p names.