prerender-v0.95.1 — @harperfast/prerender

A standalone demand tracker, changed pages ordered by demand, a cause for every bot miss, a visit ring that keeps one copy per node, and admin-started work taken out of the request's transaction (#223, #224). There is no separate 0.95.0 release.

• The demand tracker (demand.*). The visit ring moved out of render.demand into its own group. The group has the old sizing keys plus demand.enabled and demand.maxFalsePositive (0.05).
  • demandOf(url) answers with the slots visited and the estimated visit period. It answers unknown when the tracker is off or cold, or when the worst full slot's false-positive rate is above the limit.
  • The cadence ladder (render.demand) is now a consumer, and its decisions are unchanged.
  • New gauge: demand_false_positive.
• Changed pages ordered by demand (queue.ready.changedDemand, default on).
  • The probe stamps a changed page's visit period on its schedule row (RenderSchedule.demandPeriod).
  • The queue counts that row's wait in visit periods, so the most-requested changed pages render first.
  • When demand is unknown, the row orders by cadence as before.
  • queue-state reports now.changedByDemand.
• Why each miss happened. bot_miss counts one cause per origin|miss, by cause × route × bot.
  • The causes: not-found, not-modified, redirect, client-error, origin-error, passthrough, uncacheable, gated-route/bot/entity, new, unrendered, render-timeout, device, suppressed, error.
  • Per-cause distinct-URL sketches go into crawl stats under reserved ~miss: series. crawl-breadth returns misses per day and missUnion.
• Visits and sketches are no longer lost.
  • On Harper 5.2, a shared buffer nothing references is freed and comes back zeroed. coordination.getSab now holds every buffer it hands out.
  • A worker's merged bits could wait forever for a store turn. The visit ring now writes straight into one node-shared live slice, and the turn holder stores every slot that changed. Crawl stats keep per-buffer [merged, stored] generations.
• One visit-ring copy per node.
  • One node-shared union per slot, refreshed incrementally from changed rows by one worker per mergeInterval.
  • Rows record their shape (VisitFilter.shape), and every store merges the row it replaces.
  • At 16 workers and 2^23 bits: ~18 MB per node, down from ~272 MB.
• Admin-started work runs outside the request's transaction (util/detach.js).
  • This fixes Operation aborted: Database closed during transaction get operation on probe passes, sweeps, purges and sitemap walks started from the admin API. Scheduled passes were never affected.
• Unknown config keys are reported at every depth. Before, only top-level keys were checked.

Deploying

• Config, no alias.
  • Move render.demand.{bots, sliceMs, slices, bitsPerSlice, hashes, flushInterval, mergeInterval} to demand.*.
  • Set demand.enabled: true wherever render.demand.enabled is on.
  • A key left at the old path logs "Unknown configuration key" and is not applied.
  • While 0.94.0 is still loaded, it reads the moved config as having no render.demand.bots, and counts every bot until the restart.
• Schema: RenderSchedule.demandPeriod and VisitFilter.shape are additive.
• Deploy with a container restart, and before console 0.20.0.
• Ring sizing. Before relying on changed-page order, check the peak demand_fill and demand_false_positive.
  • Raising demand.bitsPerSlice reshapes the ring.
  • After a reshape the ladder rests targets at base for one slowest rung while new history accumulates.