Skip to content

v0.3.8 — named auto pools (auto = cloud, auto_home = home)

Choose a tag to compare

@oliveagle oliveagle released this 07 Nov 17:36
· 134 commits to main since this release

Named auto pools

[auto_models.<key>] declares extra rotation pools next to the default [auto_model], each answering to its own sentinel model name. A deployment can now keep the rotation sets disjoint — auto over cloud vendors, auto_home over the home GPUs — instead of one pool that mixes machines which are off half the day with cloud APIs.

  • [auto_model] stays the default pool (sentinel auto), so an existing single-pool config is unchanged. A named pool's sentinel is its table key unless the block sets name.
  • The route's provider names the pool an auto_model = true route rotates over when it matches one; otherwise the first declared pool serves it.
  • RequestCtx.auto_pool pins the pool for the whole request: the sentinel that entered the request also bounds every later failover hop, so a request can no longer drift from one rotation set into another (previously one auto pool was searched for any hop's provider name).
  • GET /v1/models lists every enabled pool sentinel (owned_by: maxllm-auto).
  • Pool names must be non-empty and unique; an auto route naming a missing or disabled pool fails validation at startup.

Verification

  • cargo test --workspace: all green (211 translate tests etc.).
  • cargo test -p maxllm-gateway --test integration: 107 passed, 4 ignored (2 new: pools keep their sets apart; a body-only auto_home sentinel is adopted into its named pool).
  • Binary: musl static, sha256 f80d23577e150935cf3a788149ac5afbd508a220d68ee5919157b6b5a37e3be5.