Skip to content

fix(selfhost): maintenance-admission gates on raw live-pending count, not runnable-now, starving the maintenance lane #4669

Description

@JSONbored

Medium severity, high confidence. Found while investigating why edge-nl-01's maintenance queue stayed backed up (24-27 pending jobs, oldest ~3h47m old) during a period of genuinely idle live-review pressure (host load 0.16/16 cores, zero foreground jobs runnable for a sustained 100s sampling window).

Context

evaluateMaintenanceAdmission (src/selfhost/maintenance-admission.ts:215-245) gates maintenance-job admission on signals.livePendingCount (live_pending_high, line 223) and signals.oldestLivePendingAgeMs (live_job_age_high, line 224). Both are computed in maintenancePressureSignals (sqlite-queue.ts:1381-1412, pg-queue.ts equivalent) as a raw count/age of every foreground-priority (priority>=8) row in pending/processing, regardless of whether it is actually due (run_after<=now).

But per-PR foreground work (agent-regate-pr) is explicitly documented as having a backlog that "is normal, expected, and can legitimately stay nonzero for long periods (staggered/rate-deferred per-PR re-reviews)" (index.ts:24-29). Every ~2-minute regate-sweep (or a full post-restart sweep) fans this out across all open PRs in every repo, which pushes livePendingCount well past the default MAINTENANCE_ADMISSION_MAX_LIVE_PENDING (5) even though almost none of that work is actually due — liveRunnableNowCount (the field the module's own doc comments identify as the correct "queue large but intentionally deferred vs. queue stuck" diagnostic, lines 78-84) stayed at 0 for the entire live-sampled window while livePendingCount bounced between 0 and 27.

Because live_pending_high is checked before maintenance_pending_high's drain-age escape (the mechanism #selfhost-maintenance-self-pin specifically added to stop the maintenance lane from self-pinning, lines 21-33), a maintenance job that should leak through the drain instead keeps failing the earlier, noisier live_pending_high check and has to wait nearly the full 4-hour maxDeferAgeMs trickle backstop. Live telemetry from edge-nl-01 confirmed this is the dominant denial reason: 73 of ~106 admission denials (69%) in an 18-minute window were live_pending_high, not host load or GitHub rate-limit pressure (host load was ~0.01 normalized the entire time). The notify-evaluate job's age matched the metrics' oldest_maintenance_pending_age_seconds almost exactly (~13.5-13.8k seconds, closing in on the 4h backstop) while sitting completely untouched across repeated 20s samples.

The same defect pattern applies to oldestLivePendingAgeMs (live_job_age_high): it's the oldest live row by created_at, not by "how long has this been due" — a deliberately staggered future-scheduled job trips it exactly the same way, and it's currently masked only because live_pending_high denies first.

Requirements

  • Gate live_pending_high on signals.liveRunnableNowCount instead of signals.livePendingCount.
  • Gate live_job_age_high on signals.oldestLiveRunnableAgeMs instead of signals.oldestLivePendingAgeMs.
  • Leave livePendingCount/oldestLivePendingAgeMs in MaintenancePressureSignals unchanged — they remain valid, correct observability gauges (gittensory_queue_live_pending, gittensory_queue_oldest_live_pending_age_seconds in server.ts), just no longer used for the admission decision.
  • No new config surface: the existing global MAINTENANCE_ADMISSION_MAX_LIVE_PENDING / MAINTENANCE_ADMISSION_MAX_LIVE_AGE_MS env knobs stay as-is — they'll simply measure the metric they were always documented to represent.
  • Regression tests (non-negotiable): a case proving a high livePendingCount with liveRunnableNowCount low/zero now admits (the exact bug), and a case proving a genuinely high liveRunnableNowCount still denies (real pressure still works). Same pair for the job-age check using oldestLiveRunnableAgeMs.

Deliverables

  • live_pending_high keyed on liveRunnableNowCount
  • live_job_age_high keyed on oldestLiveRunnableAgeMs
  • Doc comments in maintenance-admission.ts updated to state these fields now drive the admission decision, not just the dashboard diagnostic
  • Regression tests for both checks (bug-reproduction case + genuine-pressure-still-blocks case)

Expected outcome

Maintenance jobs (notifications, contributor-evidence/decision-pack builds, RAG re-indexing, retention/backfill sweeps, etc.) drain promptly whenever the box is genuinely idle, instead of being starved for hours by a normal, expected, already-accounted-for per-PR review backlog that was never actually competing for a claim slot.

References

  • src/selfhost/maintenance-admission.ts:75-105 (MaintenancePressureSignals, the runnable-vs-raw field docs), :215-245 (evaluateMaintenanceAdmission, the two checks)
  • src/selfhost/sqlite-queue.ts:1381-1412, src/selfhost/pg-queue.ts (signal computation, both backends already correct)
  • src/index.ts:24-29 (documents the agent-regate-pr backlog as normal/expected/long-lived)

Effort

S

Metadata

Metadata

Assignees

Labels

gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions