prerender-v0.96.0 — @harperfast/prerender

A negative cache for origin 404/410s, and gone targets reopened when the origin answers 200 again (#225).

• The negative cache (render.negative, route opt-in negativeCache: true). On a true miss, the origin's own 404/410 is stored, and the next crawler asking for the same dead URL is answered from it.
  • Three windows, measured from the last time the origin confirmed the status (checkedAt):
    • inside freshMs: answered from storage, and the origin is not asked. This is the only part that saves origin requests;
    • inside lifeMs: answered at once while a background re-check asks the origin. A 404/410 confirms the entry, a 200 or a redirect drops it, and a 5xx or timeout keeps it serving (stale-if-error);
    • past lifeMs: proxied, and a 404 stores again.
  • The stored body ages out too. Once the bytes are older than lifeMs, the next check that goes to the origin anyway replaces them (the re-check is a GET instead of a HEAD). No body is served more than about twice lifeMs old, at no extra origin requests.
  • It never answers a URL whose Target a sitemap lists, checked on every read and before every store (skipTargets: any also skips unlisted Targets). It never answers a bot in excludeBots. A bulk-invalidation epoch refuses entries confirmed before it, for every bot.
  • Store refusals are counted by name: has-cookie, private, no-store (ignoreNoStore for an origin that sends it on every document), staging, oversize, empty. assumeShared works as it does for the raw cache.
  • Re-checks are single-flight per key and capped per worker (maxConcurrentChecks); captures are capped like the raw cache's.
  • Counted honestly: a fresh answer is bot_serve source negative. An answer whose background re-check went to the origin is source origin, cacheStatus negative-revalidate, one per origin_fetch reason revalidate.
  • Off by default. dryRun (default on) stores and re-checks but never answers from storage, and counts would-serve (the saving) and would-serve-live (the risk: an armed cache would have answered a 404 while the origin said 200). negative_gap is the crawlers' re-ask curve to choose freshMs from.
• Reopening gone targets (render.suppression.gone.reopen, on and dry-run by default). An origin 200 for a gone-suppressed target files its recheck due now, and the render's own verdict flips it.
  • The 200 can come from a proxied bot request (any bot, even one the discovery gate refuses), a negative-cache re-check, or the target rejoining its sitemap (the arrival check used to skip every suppressed row).
  • Gone verdicts only: a noindex or canonical-mismatch page answers 200 by definition. Bounded by dedupeMs per URL (6h) and maxPerMinute per worker (60); a failed filing is retried by the next 200.
• Both outcomes of a suppressed target's render are counted, with the reason and the age since the last verdict: suppression_lifted and suppression_held. A gone target's own recheck lands in 14d+, so http-gone in the younger buckets is an early recheck, and lifted / (lifted + held) there is how often the evidence that filed it was right.
• Metrics: prerender_ops series negative_cache, negative_gap, gone_reopen, suppression_lifted, suppression_held; raw_cache is now declared (it was emitted but missing from the catalog).

Deploying

• Schema: a new database negative_cache (NegativePage, node-local, plus a replicated anchor table). It needs a worker restart per node to load.
• Nothing changes behaviour until a route sets negativeCache: true with render.negative.enabled, or render.suppression.gone.reopen.dryRun is turned off.
• With reopen on (the default), a bot-gated true 200 miss on a route that adds targets pays one detached local Target.get.
• Suggested rollout: enable the negative cache in dry run and read would-serve-live against would-serve for a few days, then arm it; arm the reopen trigger once gone_reopen would-file looks sane.