fix(scheduler): detect and recover when a job's timer stops firing

[joryirving]

Summary

• Log every scheduled run, not just failures.
• Track last-run per job and expose it on /api/health.
• Re-arm a job that ran and then stopped.

What happened

The in-app scheduler stopped running its jobs on a pod that kept serving requests normally.

pr-followup did not fire for at least 15 hours. Three PRs in misospace/llmkube-images sat at CHANGES_REQUESTED with nothing acting on them, and /api/pr-fix-queue/history returned 404 for each — the items had never been created. A manual POST to the same endpoint returned:

{"reposScanned":9,"prsScanned":4,"enqueued":35,"skipped":28,"rateLimited":false}

So the endpoint, the auth, the tracked-repo list and the ingestion gates were all fine. Only the timer was not.

Confirmed rather than inferred: the enqueue path writes a prFixHistory row on every observation, even when the evidence is already known. A single firing would have moved the count. Measured across a 17-minute window with a 15-minute interval, it stayed at 11 with the newest entry still stamped from the manual call.

Why nobody noticed

runJob logged only failures, so silence meant either "ran fine" or "never ran". An empty PR-fix queue is indistinguishable from having no work to do, and a scheduler feeding every queue in the system can stop without producing a single line of output.

Two existing tests asserted that silence explicitly (expect(deps.logs).toHaveLength(0); // 200 -> quiet). Both are updated, because that property is precisely what hid this.

What this does not do

It does not claim to fix the root cause. Why the interval stops in the standalone runtime is not established — the timers are armed with the real setInterval, the loopback base is http://127.0.0.1:${port}, auth is a bearer token whose failure would log, and nothing deletes the rows involved. Rather than guess, this detects the state and recovers from it.

The supervisor deliberately ignores a job that has never run, so a slow start is not mistaken for a dead timer; only a job that ran and then went quiet for more than two intervals is re-armed.

Verification

• npx vitest run — 2412 passed, 3 skipped.
• Two new specs: a stalled job is reported overdue and re-armed with a log line, and a never-run job is left alone.
• npm run lint clean; tsc clean apart from the pre-existing pr-fix-queue/history/route.ts error also on main.

Notes

• The boot line scheduling "X" every Yms prints before the interval is armed inside a setTimeout, so it never proved a timer existed. Commented accordingly — the supervisor's output is the real evidence.
• /api/health now carries scheduler and schedulerOverdue, which makes this alertable rather than only greppable.
• The pod was restarted to restore service before this was written; the fix is what stops it recurring silently.

[its-saffron]

AI Automated Review

Full PR review.

Analysis engine: MiniMax-M2.7@https://litellm.jory.dev/v1 (anthropic) — primary route

PR Review: fix(scheduler): detect and recover when a job's timer stops firing

Recommendation: Approve

This PR addresses a scheduler liveness bug where jobs could silently stop firing while the app continued serving requests normally. The changes add scheduler state tracking, supervisor re-arming logic, and enhanced /api/health visibility. All CI checks passed.

---

Required Checks

✅ verify route access controls are in place

The only route file modified is src/app/api/health/route.ts. This endpoint:

• Does not enforce authentication — intentionally, as health endpoints must be accessible to load balancers, Kubernetes probes, and external monitoring systems without credentials.
• The change adds scheduler and schedulerOverdue fields to the existing public GET response, which is consistent with the endpoint's existing behavior (already unauthenticated before this PR).
• No new authentication requirements or changes to auth enforcement are introduced.

Conclusion: Auth is correctly absent on this health endpoint. This matches repository convention and operational requirements.

✅ check for unintended public endpoints

The /api/health endpoint was already public. This PR:

• Adds two read-only informational fields (scheduler, schedulerOverdue) containing operational state.
• Does not enable any new mutating operations.
• Does not expose secrets or credentials.
• The scheduler field exposes job names, last-run timestamps, intervals, and overdue flags — all operational/monitoring data appropriate for a health endpoint.

Conclusion: No unintended public endpoints introduced. The health endpoint is appropriately public for monitoring purposes.

---

Standards Compliance

The AGENTS.md standards file does not define specific requirements for health endpoint access controls. The implementation follows existing patterns in the codebase where health routes are unauthenticated (as evidenced by the existing /api/health implementation).

---

Changes Summary

File	Change	Notes
src/app/api/health/route.ts	Added scheduler and schedulerOverdue to GET response	Operational telemetry for alerting
src/lib/scheduler.ts	Added schedulerState(), supervisor interval, and improved logging	Detect-and-recover for stopped timers
src/lib/scheduler.test.ts	Updated existing tests and added supervisor regression tests	2 new test cases

---

Tool Harness Findings

No tool harness output was provided in the corpus.

---

CI Status

All checks passed: Lint, Tests (2412 passed), Database integration, Coverage, Database migrations, Typecheck, Docker Build (MCP), smoke, Build, npm audit, Docker Build.

---

Unknowns / Needs Verification

None. The corpus contains sufficient evidence to verify all required checks.