Skip to content

v0.3.0

Choose a tag to compare

@janit janit released this 21 Sep 19:51
· 3 commits to main since this release

Heterogeneous local GPU scheduling. On a machine with more than one local
GPU, gator now decides where a unit may run, separately from which model
should run it.

Preference may move ordinary work; eligibility may not move heavy work.

Resource classes and eligibility

A resources file declares the backends and which of them each class may use.
Eligibility is a hard filter applied before any scoring, so no weighting can
talk a heavy unit onto the smaller card. If the card it needs is busy, the
unit queues; if the card is down, the unit waits. A property test over 500
random weightings asserts this.

Leases and a queue with no daemon

  • Backend leases are flocked JSON documents, replaced atomically. A worker
    that crashes has its lease reaped by process liveness. A lease file that will
    not parse raises an error instead of being read as "no leases held".
  • The queue is drained when something is already happening (feed,
    status/wait, and the release of the blocking lease), so nothing resident
    has to be supervised.
  • gator sched inspects and controls backends, e.g. gator sched disable.
  • Once a queued unit's only eligible backend has been down for longer than
    GATOR_BACKEND_TIMEOUT (default 1800 s), the unit is reported as
    blocked_backend. It stays queued for that backend and is never re-routed.
    Draining the queue now respects GATOR_PROBE_HEALTH, so a queued unit is no
    longer launched onto a dead backend.

Task-difficulty classifier (opt-in)

When nothing the operator configured decides a unit's class, gator can ask a
model how hard the task is. It reads the model's logprobs for two options
(heavy vs standard) from a single decode step and uses none of the text the
model writes. Units start at heavy and are demoted only above a confidence
threshold. Every failure (timeout, unreachable endpoint, unparseable reply)
keeps the unit heavy, and the record says why. Every operator setting
outranks the classifier. The endpoint passes the same trust gate as the rest of
the resources file.

gator auto plan records the classifier's recommendation on each selectable
candidate inside the hashed plan, and changing the classifier settings marks a
stored plan stale_policy.

Unchanged

Verification, merge-on-green, worktree isolation, the record and budgets. The
scheduler decides where a unit runs, not whether its result is any good.

See docs/scheduling.md. Upgrade with deno task install.