Resource-constraint-aware task dispatch and quiet-window scheduling (problem statement)

[Quidge]

Problem

Firstmate has no resource-constraint-aware task dispatch and no time-window scheduling. Task concurrency and staggering are decided entirely by firstmate's in-the-moment judgment, with no automated mechanism watching resources or gating when work starts. On a resource-limited box, running several heavy tasks at once can exhaust memory and trigger kernel OOM kills that silently destroy in-flight work.

This issue documents the problem and the constraints we've hit. It intentionally does not propose a solution.

What we observed (2026-08-05 session)

• The box is memory-constrained. ~11 GB usable RAM (16 GB physical minus a 4 GB integrated-GPU UMA carve-out — see Quidge/myrmex#25) plus 4 GB swap.
• Each agent is a heavy process. A claude/codex crew is a Node/V8 runtime holding its entire conversation plus every file it reads in RAM. Idle it's ~0.2–0.4 GB, but it can balloon — one scout reached ~7.9 GB while reading a minified JS bundle and was OOM-killed.
• The baseline is already heavy. Four resident homes (the main firstmate + three secondmates) plus any concurrent crews fill RAM and start filling swap. A heavy task on top OOM-killed a worker. The death was silent (an OOM kill leaves no app-level trace), so it looked like work was progressing when it wasn't — firstmate had to reconcile against live process state, not status lines, to catch it.
• We managed it entirely by hand. Over the session firstmate manually: staggered work to one heavy crew at a time, held tasks for load reasons, retired an idle secondmate to reclaim RAM, force-reclaimed stale swap, and continuously watched free/btop. No automation did any of this — it was all operator judgment, turn by turn.

Constraints that make it hard

• Long, resource-heavy tasks. For example, a fork-from-upstream sync runs a full ~15–19 min test suite, tying up resources for a long stretch and dominating the box while it runs.
• No resource visibility in dispatch. Nothing watches memory/CPU and gates when a task starts; firstmate is the scheduler, by judgment, every time. Getting it wrong OOMs the box; being too conservative wastes real headroom (the captain could see spare capacity in btop that the manual staggering wasn't using).
• A natural quiet window goes unused. The captain sleeps ~8–9 hours nightly — an obvious window to run heavy/long work unattended — but there is no mechanism to defer work to that window or to run it automatically then.

What firstmate already has (relevant primitives)

• A first-class load hold kind: tasks-axi hold <id> --kind load records a task held for resource/load reasons. Today it is only a manual marker — nothing auto-releases it when resources free up.
• A future hold kind and date gates (hold --until YYYY-MM-DD) at day granularity.
• AFK mode (/afk): a daemon that supervises overnight and batches escalations. It supervises work already in flight; it does not appear to proactively dispatch newly-eligible queued work, and it has no time-of-day granularity.

Open questions (undecided — not prescribing an answer)

• Should resource-awareness be automated (something that watches memory and gates or releases dispatch), or remain firstmate's judgment with better tooling to inform it?
• How should "defer to a quiet window / run overnight while I'm away" be expressed on a task and honored?
• Should the AFK daemon gain the ability to dispatch newly-eligible queued work, or is a separate scheduler cleaner?
• How should resource limits and headroom be modeled — memory thresholds, per-task resource hints, a max-concurrency notion, something else?

Cross-reference: the underlying RAM constraint (and its partial reclaim via the BIOS iGPU setting) is tracked in Quidge/myrmex#25.