Skip to content

ramjet v0.7.0

Latest
Latest

Choose a tag to compare

View all tags
@nessie993 nessie993 released this 29 Sep 09:08
· 14 commits to main since this release
v0.7.0
2e124d8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Ramjet v0.7.0 routes one fleet across many nodes, and across the data-parallel attention ranks inside each node.

Highlights:

  • RJ_ROUTE_AFFINITY_BASIS=relative. Each replica is scored against the warmest serving peer of its model.
    • marginal (#294) used the least-warm peer as its floor. Beyond two replicas that peer is usually cold, so marginal fell back to absolute.
    • In the fleet simulation at 40 replicas, relative kept 90% of agent turns on the replica holding their session, against 51% for absolute and marginal.
    • At two replicas it makes the same decisions as marginal.
  • RJ_TOPOLOGY_FILE. The fleet is described as named nodes and their replicas instead of index-aligned comma lists. Node names appear on ramjet_upstream_info{upstream,node} and in /health.
  • SGLang data-parallel attention ranks as upstreams (RJ_UPSTREAM_DP_RANKS, or "dp_ranks": N on a topology replica). Requests are pinned with routed_dp_rank, so prefix affinity chooses the rank. On GLM-5.3 DP8 (FP8 KV, 8×H200) with 16 agent developers, this gave 71.9 turns/min at 92.9% cached prompt, against 43.0 turns/min at 65.6% for SGLang's own round robin.
  • Fleet resilience:
    • Failover tries other nodes before a failed replica's siblings, within RJ_ROUTE_MAX_ATTEMPTS.
    • A refused connection fences all of that engine's ranks.
    • RJ_UPSTREAM_RANK_PROBE=on detects a wedged DP rank that /health still reports as up.
    • RJ_UPSTREAM_CONNECT_TIMEOUT_MS sets the connect budget, and healthy upstreams are probed concurrently.
  • Pooled upstream connections now expire after 4s idle, below the engines' 5s keep-alive (#296). This is the one changed default.
  • bench/agent_swarm_bench.py, an open-loop coding-agent fleet generator for routing A/B tests (#295).

Every other new behaviour is opt-in; existing deployments route as before. docs/multi-node.md describes the multi-node configuration.

Immutable images:

  • Load balancer: ghcr.io/helixml/ramjet:v0.7.0@sha256:dca028638314ca3171120532a075faaa70483e1494dd3d04bddc4db4eb88c01d
  • Snapshot companion: ghcr.io/helixml/ramjet:companion-v0.7.0@sha256:562b63d16253cb1cc9f823fe41bf7146c448d3de806f49db50f5421cc4342880

Qualification. This release was tagged after the full CI pipeline passed. The node06 acceptance step in RELEASE.md was not performed. The routing features were exercised on 8×H200 GLM-5.3 test nodes, not on node06.

Rollback: the last node06-qualified load balancer is ghcr.io/helixml/ramjet:v0.6.2@sha256:53047a816c8ae1dbade7e27b86d2e3eda40cfeaa48826ae06bf2ac1d35a27cc6.

🤖 Generated with Claude Code

Assets 2
Loading