Skip to content

v0.6.2

Latest
Latest

Choose a tag to compare

View all tags
@nessie993 nessie993 released this 25 Sep 15:23
· 23 commits to main since this release
v0.6.2
962b7b2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Ramjet v0.6.2 adds an opt-in long-prompt lane and ships the GLM-5.3 prefix-cache fixes for node06.

Highlights:

  • Long-prompt lane. RJ_ROUTE_LONG_PROMPT_BYTES and RJ_ROUTE_LONG_PROMPT_UPSTREAMS (lane/- per upstream) confine requests at or above a body-size threshold to designated replicas, so one very long prefill can't evict a peer's prefix cache or stall its requests. If no lane member is serving, the request routes normally. Adds ramjet_route_long_prompt_total{upstream,outcome} and route-journal v12. Unset or 0 turns it off.
  • GLM-5.3 prefix cache (node06 deployment). The hybrid model's 28-slot linear-attention state pool, not KV space, was evicting every other session's prefix whenever a ~310k-token prompt arrived. The replicas now cap cached states per radix path and add a HiCache host tier:
    • six 20k-token sessions stay 99.6% cached across a 308k-token prompt (previously 0/6);
    • twelve sessions fit on the device (previously none);
    • host restores take 0.43 s instead of 3.4 s.
  • GLM-5.3 SwiGLU clamp. The derived SGLang image routes swiglu_limit = 10.0 into the SM120 W4A16 routed-expert kernel; the pinned SGLang and FlashInfer both dropped it. Full GSM8K went from 96.29% to 96.44%, and the agent corpus passes 5/5.
  • System One API routing (RJ_UPSTREAM_APIS, /v1/systemone) and the Kev runtime recipe.

Immutable images:

  • Load balancer: ghcr.io/helixml/ramjet:v0.6.2@sha256:53047a816c8ae1dbade7e27b86d2e3eda40cfeaa48826ae06bf2ac1d35a27cc6
  • Snapshot companion: ghcr.io/helixml/ramjet:companion-v0.6.2@sha256:5c7ee804d5d8f5032335cc0184b9e06212d7eb1779b508b0d29f8f0124c91864

These exact load-balancer bytes were qualified on node06 before tagging:

  • the guarded canary rollout passed, with 4/4 upstreams healthy and both API families exercised;
  • a 654 KB request was routed through the lane to glm53sm120-c and a short request went to glm53sm120-b;
  • no LB errors or 5xx;
  • a Helix end-to-end run of the Dubai broker intake suite (11 cases) on warm caches reached a 93.1% prompt-cache hit ratio, a 98.9% hit on each session's first large call, and a p50 TTFT of 1.10 s.

Immediate rollback retained on node06: ghcr.io/helixml/ramjet:systemone-6b025f7-8fa31d5cec1d@sha256:bcd132f97171f6203715e04f0fc5ab3784a8b8e8cb656e7bf74c1c39b9884f93 (ds4-loadbalancer-rollback-20260925T151249Z). The lane alone can be turned off with RJ_ROUTE_LONG_PROMPT_BYTES=0.

Assets 2
Loading