Repository navigation
v0.3.0
Heterogeneous local GPU scheduling. On a machine with more than one local
GPU, gator now decides where a unit may run, separately from which model
should run it.
Preference may move ordinary work; eligibility may not move heavy work.
Resource classes and eligibility
A resources file declares the backends and which of them each class may use.
Eligibility is a hard filter applied before any scoring, so no weighting can
talk a heavy unit onto the smaller card. If the card it needs is busy, the
unit queues; if the card is down, the unit waits. A property test over 500
random weightings asserts this.
Leases and a queue with no daemon
- Backend leases are
flocked JSON documents, replaced atomically. A worker
that crashes has its lease reaped by process liveness. A lease file that will
not parse raises an error instead of being read as "no leases held". - The queue is drained when something is already happening (
feed,
status/wait, and the release of the blocking lease), so nothing resident
has to be supervised. gator schedinspects and controls backends, e.g.gator sched disable.- Once a queued unit's only eligible backend has been down for longer than
GATOR_BACKEND_TIMEOUT(default 1800 s), the unit is reported as
blocked_backend. It stays queued for that backend and is never re-routed.
Draining the queue now respectsGATOR_PROBE_HEALTH, so a queued unit is no
longer launched onto a dead backend.
Task-difficulty classifier (opt-in)
When nothing the operator configured decides a unit's class, gator can ask a
model how hard the task is. It reads the model's logprobs for two options
(heavy vs standard) from a single decode step and uses none of the text the
model writes. Units start at heavy and are demoted only above a confidence
threshold. Every failure (timeout, unreachable endpoint, unparseable reply)
keeps the unit heavy, and the record says why. Every operator setting
outranks the classifier. The endpoint passes the same trust gate as the rest of
the resources file.
gator auto plan records the classifier's recommendation on each selectable
candidate inside the hashed plan, and changing the classifier settings marks a
stored plan stale_policy.
Unchanged
Verification, merge-on-green, worktree isolation, the record and budgets. The
scheduler decides where a unit runs, not whether its result is any good.
See docs/scheduling.md. Upgrade with deno task install.