orb(queue): live_job_age_high fires on a normally-running job with no drain escape, collapsing the whole maintenance lane (watchdog included) to a 4-hour trickle

[JSONbored]

Problem

Two defects in maintenance admission that compound: the lane shuts itself off during normal operation,
and the denial path amplifies the very pressure it measures.

1. live_job_age_high fires on a normally-running job, and has no drain escape

oldestLiveRunnableAgeMs is now - MIN(created_at) over foreground rows that are status='processing'
or due (src/selfhost/pg-queue.ts:443-449, :472) — it includes a job that is currently running
normally. A single in-flight AI review (routinely minutes) pushes that value past
DEFAULT_MAX_LIVE_JOB_AGE_MS = 2 * 60_000 (src/selfhost/maintenance-admission.ts:135) for its whole
duration, and:

if (signals.oldestLiveRunnableAgeMs > config.maxLiveJobAgeMs) return { admit: false, reason: "live_job_age_high" };

Unlike maintenance_pending_high, this branch has no drain escape (:253-263 gives the drain only to
the pending-count branch). The only way out is the 4-hour trickle_max_defer_age backstop.

The metric conflates "a job has been running for 2 minutes" with "live work is starved".

Impact: ops-alerts, sweep-liveness-watchdog, notify-deliver, prune-retention,
backlog-convergence-sweep, and reconcile-open-prs degrade from 30-min/hourly cadence to once per 4
hours on any instance with near-continuous review traffic. The watchdog and the alerter that would report
"the box is overloaded / the sweep stopped" are themselves in the lane the overload throttles — the
monitor dies with the thing it monitors. This is also why the retention job in #9083 barely runs.

Fix: compute oldestLiveRunnableAgeMs from unclaimed due-pending rows only (exclude
status='processing'), or give this branch the same bounded drain escape maintenance_pending_high has.

2. Each denied job re-runs four full aggregate scans

Every claimed maintenance job calls await maintenancePressureSignals(Date.now())
(src/selfhost/pg-queue.ts:1187-1193) → four separate un-cached aggregate queries over _selfhost_jobs
(:438-479). A denial returns true from processOne, so the pump loop
(while (!shuttingDown && (await processOne())), :1489) immediately claims the next due maintenance row
and repeats.

N due maintenance rows in a burst ⇒ 4N sequential aggregate scans plus N UPDATEs plus N spans, in one
tight loop. The maintenance count (is_maintenance=1) has no supporting index — the DDL at :200-204
indexes status,priority,claim_sort_key,run_after, job_key,status, status,foreground_lane,run_after,
and status,dead_at,id.

This is a positive feedback loop: more denials → more scans → higher DB and host CPU → higher
hostLoadAvg1PerCore → more denials. All on the same Postgres pool (PGPOOL_MAX default 10,
src/selfhost/queue-common.ts:737) shared with every HTTP handler and review job.

Secondary, same area: claimNextForegroundLane (:1009-1011) does a serialized
UPDATE … WHERE id='singleton' on every claim attempt including idle 1s poll ticks, so all 8 pumps'
claim paths serialize on one row and the table accrues a dead tuple per attempt, 24/7.

Fix: memoize maintenancePressureSignals for a short TTL (1–2s) or once per pump-drain pass; fold the
four queries into one FILTER-based statement; add a partial index on (is_maintenance, status).

Test Coverage Requirements

99%+ patch coverage, branch-counted. For (1): both arms with a long-running processing row present. For
(2): assert one signals computation per drain pass, not per denied job.

Links & Resources

• src/selfhost/maintenance-admission.ts ~43-71, ~135, ~245-263; src/selfhost/pg-queue.ts ~200-204,
  ~438-479, ~1009-1011, ~1187-1193, ~1489; src/selfhost/queue-common.ts ~737
• orb(db): 3.4 GB in one month — no retention target has a usable index, and four caches have no delete path at all #9083 (DB growth/retention — this is why the retention job barely runs)

maintainer-only — admission control and DB load.

[linear-code]

JSO-6327