Foretoken 0.0.5

What's changed

Deployment and source development

• Run independently updatable named deployments in the same Kubernetes namespace. Each instance owns its controller and serving resources; the shared platform, infrastructure and storage permissions remain administrator-managed. Deployment, status, stop, deletion, benchmarking and profiling use the same instance name.
• Release artifacts now separate runtime environment images from Foretoken applications. The application archive supplies the controller, frontend and model-server programs, while images contain their runtime dependencies. Source updates reuse unchanged environments and publish only the affected applications for the selected instance.
• Source builds automatically import matching published BuildKit caches. Native dependency compilation, including Mooncake, can be reused across fresh development environments. Build-worker counts no longer invalidate compilation caches. Parallel preparation, native node-content reuse and batched application publication reduce repeated build and delivery work.
• Share downloaded model files between instances while keeping mutable runtime and profiling caches separate. foretoken stop releases serving resources and retains configuration and model references; foretoken delete retires execution before releasing references. Managed model files are collected after their last service reference is removed.
• New installations share one HTTP Gateway by default, with per-instance Host routing. Installation prepares authorized directory bindings and reuses suitable cluster infrastructure. k3d directory discovery now handles unrelated file bind mounts correctly.

Client APIs, admission and online configuration

• Responses supports saved results, continuation by response ID, Conversations, background generation, cancellation and resumable background event streams. Saved Chat Completions supports retrieval, listing, metadata updates and deletion. History is shared across frontend replicas in a temporary in-memory cache; applications needing permanent retention should retain their own copy.
• Configure shared waiting limits and deadlines, per-caller waiting and concurrency allowances, role priorities and permitted model Pools. Model instances can separately limit accepted unfinished work. Caller rules and prefix-cache isolation use verified identities supplied by an authentication gateway.
• Routing, admission, prompt limits, request timeouts and frontend logging can update online. Model environment variables support Kubernetes Secret, ConfigMap, Pod-field and resource references. Environment or prefix-cache-isolation changes replace the affected model instances through normal deployment updates.

Routing, KV reuse and autoscaling

• Mooncake and NIXL can reuse KV retained by an earlier Decode instance during a later Prefill, including across frontend and Prefill instances. ModelService.spec.kvTransfer selects the connector and optional transport override; normal configurations select a compatible transport automatically. Retained reuse supports full-attention text with single-node replicas, without LoRA or pipeline/context parallelism; NIXL additionally requires matching Prefill/Decode tensor-parallel sizes.
• Override routing independently for a model and its Encoder, Prefill and Decode stages. Cache-aware routing exposes weights for GPU-local and restored-prefix reuse alongside request pressure.
• Add session_affinity, endpoint_attribute and throughput scorers. They respectively use request session_id, supported numeric endpoint observations, and weighted request lengths to choose an endpoint.
• Add per-data-parallel-engine token_load autoscaling with throughput, latency and explicit-threshold modes. Missing observations hold capacity, and existing replica bounds, stabilization and draining continue to apply.

Inference runtimes and hardware

• The default NVIDIA environment advances to vLLM 0.31, with its matching CUDA/PyTorch runtime and the complete Mooncake native package, including transfer, storage, expert-parallel and process-group components.
• Installation discovers supported RDMA resources and prepares compatible network attachments while retaining externally managed allocations. Multi-node replicas can use administrator-provided Volcano queues, Kueue quota admission, or both.
• MetaX source integration includes GLM speculative-decoding, mHC compilation, graph-startup and communication fixes. This release's prebuilt runtime images cover NVIDIA; the MetaX image is deferred, and MetaX source deployment remains available.

Evaluation, recovery and RSI

• Benchmark requests and run state are retained in SQLite. Interrupted sweeps and searches preserve completed work, and completed measurements can finish plotting or publication without starting a model service again. Unfinished measurements restart from warmup.
• W&B publication runs in the background with local results and upload logs retained. Offline results can move between machines and be uploaded later. Saved-result plotting accepts multiple directories and a --base reference for measured values and relative comparisons.
• Experiment output groups commands, source context, measurements and iteration notes. RSI playbooks provide task templates, role guidance, GPU-utilization collection, a research library and a reusable deploy–measure–update workflow.
• Video benchmarking uses foretoken perf --video. A separate health endpoint is checked only when supplied with --health-url.

Operations and reliability

• Evaluation dependencies pin LiteLLM to 1.106.2 following its publishing compromise.
• Consistent diagnostics and request IDs improve cross-component tracing. Admission dashboards and persistent logs cover more serving, preparation, video and cleanup workloads.
• Profiling recovery retains stop observations across process replacement, and closing a profile viewer releases its owned processes.
• Deployment and cancellation fixes preserve the original failure during cleanup, retain selected applications across recovery, avoid replacing unchanged model processes and maintain ownership of accepted requests until execution ends.

Artifacts

Artifact	Version or location
Python package	foretoken==0.0.5 on PyPI
Helm Chart	oci://ghcr.io/shiweijiezero/foretoken/charts/foretoken, version 0.0.5
Application archive	foretoken-applications-0.0.5-linux-amd64.tar.gz, attached to this release
Control-plane environment	ghcr.io/shiweijiezero/foretoken/control-plane-environment:0.0.5
Frontend environment	ghcr.io/shiweijiezero/foretoken/frontend-environment:0.0.5
NVIDIA model-server environment	ghcr.io/shiweijiezero/foretoken/model-server-environment:0.0.5

The application archive and runtime images target Linux amd64. Matching build caches are published alongside the environments and selected automatically by source builds. Omni/video serving uses its separately configured runtime.

Getting started and upgrading

Install the CLI and get the matching examples:

git clone --branch v0.0.5 https://github.com/shiweijiezero/foretoken.git
cd foretoken
uv venv --python 3.12 --seed  # With pip: python3 -m venv .venv (Python 3.11+)
source .venv/bin/activate
uv pip install foretoken==0.0.5  # Or: pip install foretoken==0.0.5

Follow the 0.0.5 Quick Start for cluster setup, deployment and a first request. On a prepared cluster, foretoken install installs the published platform. To develop from the matching checkout, use foretoken install -e ., followed by foretoken deploy examples/quickstart/qwen3-0.6b -e . for subsequent source updates.

Important changes for existing installations:

• Upgrade the shared platform and CRDs before operating new named deployments. Retire older namespace-wide controllers through their existing installation owner before instance controllers operate in that namespace.
• Service commands use the current kubeconfig namespace. Without --name, the instance name comes from FrontendService.metadata.name; the single-model Quick Start uses qwen3-0-6b. Pass an existing instance name explicitly when retaining it. Ordinary deployment uses installed applications; source updates require explicit -e PATH.
• New managed Gateways listen on HTTP port 8081 and route by Host. Use both values returned by foretoken endpoint and foretoken endpoint --host. Select --frontend-mode local when separate LoadBalancer Services are intended.
• Move P/D transport choices to ModelService.spec.kvTransfer. Remove old runtime.vllm.pd and modelDistribution.modelexpress / dragonfly enablement blocks from newly supplied values files. The installer migrates saved platform values. Check custom image and application settings against the new environment-image/application-archive layout.
• Replace foretoken perf video with foretoken perf --video. Async video submission uses the public {model, request} envelope instead of modelServiceRef.
• Each sweep invocation creates a new result directory; --experiment-name supplies a prefix. Keep records.sqlite3 and the original directories referenced by resumed runs, preserving their relative layout when moving results.

Platform and source operations require Helm 4.2.4 or later. Python 3.11+ and Kubernetes 1.29+ remain the documented minimums. The host driver must support the selected inference runtime.

Known issues

In 0.0.5, runtime-image pulls consume the model preparation Job's spec.timeouts.startup budget. On uncached nodes, a slow pull can exhaust the Quick Start's five-minute budget before the preparation process starts; allow more time through that model setting and the CLI's --timeout. Automatic registry selection can also choose a proxy whose manifest is reachable but whose image layers are unavailable. An explicit runtime.vllm.image Helm value selects the desired runtime source.

These issues are fixed in 0.0.7, which also improves workload recovery and source-build reuse.

Thanks

• @RSL459 — session-affinity, endpoint-attribute and weighted request-length routing scorers.
• @xinghuanliukong — per-engine token-load autoscaling and scheduler-demand integration.
• @shiweijiezero — deployment and storage lifecycle integration, stateful APIs, admission, KV reuse, source-development workflows, benchmark recovery and RSI tooling.

Full changelog

Changes from v0.0.4 to v0.0.5