fix(selfhost): gate maintenance admission on genuine live pressure, not raw pending count#4674
Merged
Merged
Conversation
…ot raw pending count (#4669) evaluateMaintenanceAdmission's live_pending_high/live_job_age_high checks used livePendingCount/oldestLivePendingAgeMs, which include work deliberately deferred to the future (e.g. agent-regate-pr's normal, staggered per-PR backlog). That let a routine, expected backlog starve the maintenance lane for hours even when the box was otherwise idle, since the checks never reached the drain escape meant to prevent exactly this. Switch both checks to the runnable-now/processing signal they were always meant to use, and broaden that signal to also count actively processing jobs as genuine current pressure (not just due-and-unclaimed ones).
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4674 +/- ##
=======================================
Coverage 94.13% 94.13%
=======================================
Files 434 434
Lines 38541 38541
Branches 14054 14054
=======================================
Hits 36281 36281
Misses 1601 1601
Partials 659 659
🚀 New features to boost your workflow:
|
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evaluateMaintenanceAdmission'slive_pending_high/live_job_age_highchecks gated onlivePendingCount/oldestLivePendingAgeMs— a raw pending/processing count and created-at age that include work deliberately deferred to the future (e.g.agent-regate-pr's normal, expected, staggered per-PR backlog,index.ts:24-29). On edge-nl-01 this let a routine backlog (not real load: host load was ~0.01/core, zero foreground jobs runnable for a sustained 100s sample) starve the maintenance lane for hours — 69% of admission denials in an 18-minute window werelive_pending_high, and themaintenance_pending_highdrain escape built specifically to prevent lane self-pinning never got a chance to fire because this check denies first.liveRunnableNowCount/oldestLiveRunnableAgeMs), and broadens that signal's SQL in both queue backends (sqlite + pg) to also count activelyprocessingforeground jobs as genuine current pressure, not just due-and-unclaimedpendingones — aprocessingjob is real, in-flight resource use and arguably more legitimate pressure than a merely-due-but-waiting one.MAINTENANCE_ADMISSION_MAX_LIVE_PENDING/MAINTENANCE_ADMISSION_MAX_LIVE_AGE_MSenv knobs are unchanged — they now simply measure what their doc comments always said they measured.Closes #4669
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
src/selfhost/{maintenance-admission,sqlite-queue,pg-queue}.tswith no UI, MCP, OpenAPI, wrangler binding, or workflow surface, soactionlint/test:workers/build:mcp/test:mcp-pack/ui:*are not applicable.test:coveragewas run targeted (vitest run test/unit/selfhost-{maintenance-admission,sqlite-queue,pg-queue}.test.ts --coverage, 330 tests, all green) rather than the full unsharded suite;maintenance-admission.ts's only reported uncovered line (106) is a pre-existing JSDoc comment line outside this diff, unrelated to the change.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. (N/A — no visible UI change)UI Evidence
N/A — no UI/frontend/docs-visible change; this is an internal self-host queue admission fix.
Notes
/metricsendpoint and its Postgres_selfhost_jobstable during the investigation; details are in fix(selfhost): maintenance-admission gates on raw live-pending count, not runnable-now, starving the maintenance lane #4669.