Foretoken 0.0.7

What's changed

Reuse node images during source development

• Source builds avoid downloading the full runtime again when its layers are already present on the node. Missing content is fetched automatically.
• Automatic GitHub and Go download-source selection no longer triggers native Mooncake recompilation. Release builds and source deployments share the same cached native build inputs.
• The Quick Start installs the published CLI and platform by default; code changes and parallel experiments use the source deployment workflow.

Preserve startup budgets through workload recovery

• Availability observations now identify the Deployment or LeaderWorkerSet they belong to. A replacement workload waits for its own first model container before starting its budget; Pods retiring from an older workload no longer affect that clock.
• The current workload keeps one unavailable interval across container restarts and member replacement. Recreating the workload starts a separate interval even if the preceding workload never became available.

Artifacts

This release provides foretoken==0.0.7 on PyPI, Helm Chart 0.0.7, the application archive attached below, and Linux amd64 control-plane, frontend and NVIDIA model-server environments with matching source-build caches. The inference runtime remains vLLM 0.31 with the complete Mooncake package. The MetaX runtime image remains deferred; source deployment is available.

Upgrading

Upgrade the CLI in its active environment:

uv pip install --upgrade foretoken==0.0.7  # Or: pip install --upgrade foretoken==0.0.7

The platform administrator runs foretoken install with the existing registry and custom values options, then redeploys the instances that should adopt the new controller. The platform upgrade adds ModelGroup.status.observedWorkloadUID. Previously unavailable Groups receive one fresh startup window when they first adopt workload identity; subsequent recovery retains the recorded window.

For source installations, select this release in a clean checkout, then update the platform:

git fetch origin tag v0.0.7
git switch --detach v0.0.7
foretoken install -e .

Redeploy with foretoken deploy PATH --name INSTANCE -e ., using the instance's existing Kustomize directory and name. Retain any registry and custom values options used by the installation.

Thanks

• @shiweijiezero — native content reuse, workload-recovery fixes and end-to-end validation.

Full changelog

Changes from v0.0.6 to v0.0.7

Foretoken 0.0.6

What's changed

Cold installation and model startup

• Model preparation and serving startup now receive their full spec.timeouts.startup budget after the first model container starts. Scheduling, runtime-image downloads and init-container setup no longer consume that budget. Preparation remains bounded by Kubernetes' native Job deadline, including Pod retries.
• Preparation failures retain Kubernetes' original deadline reason while Pods terminate, instead of being reported as an unspecified container failure.
• Platform installation prefers an accessible official OCI registry. Public proxies remain available when the origin cannot be reached, and explicit registry or image settings are preserved. A faster manifest response alone no longer moves an installation to another runtime-image source.
• Source and release builds normalize native dependency input permissions, so equivalent checkouts reuse Mooncake compilation even when shared directories use a different umask. Source files keep their local permissions.

Benchmark startup and recording

• foretoken perf parses help and arguments before loading measurement backends. Fixed prompts and local data avoid initializing EvalScope's unrelated benchmark registry and Hugging Face dataset readers; random and remote-dataset workloads still use their existing native implementations.
• Video help also avoids dataset initialization. HTTP, trace, SLO and video execution retain their existing commands and measurement clocks, and the normal installation still includes the complete dependency set.
• Diagnostics now respect SDK loggers that intentionally keep their records private. This fixes W&B recording failures during shutdown while retaining both Foretoken's run.log and W&B's native diagnostics and recordings.

Artifacts

This release provides foretoken==0.0.6 on PyPI, Helm Chart 0.0.6, foretoken-applications-0.0.6-linux-amd64.tar.gz, and these Linux amd64 environments:

ghcr.io/shiweijiezero/foretoken/control-plane-environment:0.0.6
ghcr.io/shiweijiezero/foretoken/frontend-environment:0.0.6
ghcr.io/shiweijiezero/foretoken/model-server-environment:0.0.6

Matching source-build caches are published with the environments. The NVIDIA inference environment retains vLLM 0.31 and the complete Mooncake package from 0.0.5. The MetaX runtime image remains deferred; source deployment is available.

Upgrading

Upgrade the CLI in its active Python environment:

uv pip install --upgrade foretoken==0.0.6  # Or: pip install --upgrade foretoken==0.0.6

The platform administrator then runs foretoken install, retaining any existing custom values or registry options. This updates the platform template and the controller permission needed to set preparation deadlines. Redeploy each instance that should adopt the new controller and defaults, using its existing configuration and instance name.

For source installations, update to the matching source and use foretoken install -e ., followed by foretoken deploy PATH --name INSTANCE -e .. Existing deployments retain their selected applications until explicitly redeployed.

See the 0.0.6 Quick Start for a new installation.

Known issues

Workload replacement can inherit an incorrect startup deadline. Source builds can repeat runtime downloads or Mooncake compilation even when compatible content is available. These issues are fixed in 0.0.7.

Thanks

• @shiweijiezero — cold-start lifecycle fixes, benchmark startup and logging improvements, and release validation.

Full changelog

Changes from v0.0.5 to v0.0.6

Foretoken 0.0.5

What's changed

Deployment and source development

• Run independently updatable named deployments in the same Kubernetes namespace. Each instance owns its controller and serving resources; the shared platform, infrastructure and storage permissions remain administrator-managed. Deployment, status, stop, deletion, benchmarking and profiling use the same instance name.
• Release artifacts now separate runtime environment images from Foretoken applications. The application archive supplies the controller, frontend and model-server programs, while images contain their runtime dependencies. Source updates reuse unchanged environments and publish only the affected applications for the selected instance.
• Source builds automatically import matching published BuildKit caches. Native dependency compilation, including Mooncake, can be reused across fresh development environments. Build-worker counts no longer invalidate compilation caches. Parallel preparation, native node-content reuse and batched application publication reduce repeated build and delivery work.
• Share downloaded model files between instances while keeping mutable runtime and profiling caches separate. foretoken stop releases serving resources and retains configuration and model references; foretoken delete retires execution before releasing references. Managed model files are collected after their last service reference is removed.
• New installations share one HTTP Gateway by default, with per-instance Host routing. Installation prepares authorized directory bindings and reuses suitable cluster infrastructure. k3d directory discovery now handles unrelated file bind mounts correctly.

Client APIs, admission and online configuration

• Responses supports saved results, continuation by response ID, Conversations, background generation, cancellation and resumable background event streams. Saved Chat Completions supports retrieval, listing, metadata updates and deletion. History is shared across frontend replicas in a temporary in-memory cache; applications needing permanent retention should retain their own copy.
• Configure shared waiting limits and deadlines, per-caller waiting and concurrency allowances, role priorities and permitted model Pools. Model instances can separately limit accepted unfinished work. Caller rules and prefix-cache isolation use verified identities supplied by an authentication gateway.
• Routing, admission, prompt limits, request timeouts and frontend logging can update online. Model environment variables support Kubernetes Secret, ConfigMap, Pod-field and resource references. Environment or prefix-cache-isolation changes replace the affected model instances through normal deployment updates.

Routing, KV reuse and autoscaling

• Mooncake and NIXL can reuse KV retained by an earlier Decode instance during a later Prefill, including across frontend and Prefill instances. ModelService.spec.kvTransfer selects the connector and optional transport override; normal configurations select a compatible transport automatically. Retained reuse supports full-attention text with single-node replicas, without LoRA or pipeline/context parallelism; NIXL additionally requires matching Prefill/Decode tensor-parallel sizes.
• Override routing independently for a model and its Encoder, Prefill and Decode stages. Cache-aware routing exposes weights for GPU-local and restored-prefix reuse alongside request pressure.
• Add session_affinity, endpoint_attribute and throughput scorers. They respectively use request session_id, supported numeric endpoint observations, and weighted request lengths to choose an endpoint.
• Add per-data-parallel-engine token_load autoscaling with throughput, latency and explicit-threshold modes. Missing observations hold capacity, and existing replica bounds, stabilization and draining continue to apply.

Inference runtimes and hardware

• The default NVIDIA environment advances to vLLM 0.31, with its matching CUDA/PyTorch runtime and the complete Mooncake native package, including transfer, storage, expert-parallel and process-group components.
• Installation discovers supported RDMA resources and prepares compatible network attachments while retaining externally managed allocations. Multi-node replicas can use administrator-provided Volcano queues, Kueue quota admission, or both.
• MetaX source integration includes GLM speculative-decoding, mHC compilation, graph-startup and communication fixes. This release's prebuilt runtime images cover NVIDIA; the MetaX image is deferred, and MetaX source deployment remains available.

Evaluation, recovery and RSI

• Benchmark requests and run state are retained in SQLite. Interrupted sweeps and searches preserve completed work, and completed measurements can finish plotting or publication without starting a model service again. Unfinished measurements restart from warmup.
• W&B publication runs in the background with local results and upload logs retained. Offline results can move between machines and be uploaded later. Saved-result plotting accepts multiple directories and a --base reference for measured values and relative comparisons.
• Experiment output groups commands, source context, measurements and iteration notes. RSI playbooks provide task templates, role guidance, GPU-utilization collection, a research library and a reusable deploy–measure–update workflow.
• Video benchmarking uses foretoken perf --video. A separate health endpoint is checked only when supplied with --health-url.

Operations and reliability

• Evaluation dependencies pin LiteLLM to 1.106.2 following its publishing compromise.
• Consistent diagnostics and request IDs improve cross-component tracing. Admission dashboards and persistent logs cover more serving, preparation, video and cleanup workloads.
• Profiling recovery retains stop observations across process replacement, and closing a profile viewer releases its owned processes.
• Deployment and cancellation fixes preserve the original failure during cleanup, retain selected applications across recovery, avoid replacing unchanged model processes and maintain ownership of accepted requests until execution ends.

Artifacts

Artifact	Version or location
Python package	foretoken==0.0.5 on PyPI
Helm Chart	oci://ghcr.io/shiweijiezero/foretoken/charts/foretoken, version 0.0.5
Application archive	foretoken-applications-0.0.5-linux-amd64.tar.gz, attached to this release
Control-plane environment	ghcr.io/shiweijiezero/foretoken/control-plane-environment:0.0.5
Frontend environment	ghcr.io/shiweijiezero/foretoken/frontend-environment:0.0.5
NVIDIA model-server environment	ghcr.io/shiweijiezero/foretoken/model-server-environment:0.0.5

The application archive and runtime images target Linux amd64. Matching build caches are published alongside the environments and selected automatically by source builds. Omni/video serving uses its separately configured runtime.

Getting started and upgrading

Install the CLI and get the matching examples:

git clone --branch v0.0.5 https://github.com/shiweijiezero/foretoken.git
cd foretoken
uv venv --python 3.12 --seed  # With pip: python3 -m venv .venv (Python 3.11+)
source .venv/bin/activate
uv pip install foretoken==0.0.5  # Or: pip install foretoken==0.0.5

Follow the 0.0.5 Quick Start for cluster setup, deployment and a first request. On a prepared cluster, foretoken install installs the published platform. To develop from the matching checkout, use foretoken install -e ., followed by foretoken deploy examples/quickstart/qwen3-0.6b -e . for subsequent source updates.

Important changes for existing installations:

• Upgrade the shared platform and CRDs before operating new named deployments. Retire older namespace-wide controllers through their existing installation owner before instance controllers operate in that namespace.
• Service commands use the current kubeconfig namespace. Without --name, the instance name comes from FrontendService.metadata.name; the single-model Quick Start uses qwen3-0-6b. Pass an existing instance name explicitly when retaining it. Ordinary deployment uses installed applications; source updates require explicit -e PATH.
• New managed Gateways listen on HTTP port 8081 and route by Host. Use both values returned by foretoken endpoint and foretoken endpoint --host. Select --frontend-mode local when separate LoadBalancer Services are intended.
• Move P/D transport choices to ModelService.spec.kvTransfer. Remove old runtime.vllm.pd and modelDistribution.modelexpress / dragonfly enablement blocks from newly supplied values files. The installer migrates saved platform values. Check custom image and application settings against the new environment-image/application-archive layout.
• Replace foretoken perf video with foretoken perf --video. Async video submission uses the public {model, request} envelope instead of modelServiceRef.
• Each sweep invocation creates a new result directory; --experiment-name supplies a prefix. Keep records.sqlite3 and the original directories referenced by resumed runs, preserving their relative layout when moving results.

Platform and source operations require Helm 4.2.4 or later. Python 3.11+ and Kubernetes 1.29+ remain the documented minimums. The host driver must support the selected inference runtime.

Known issues

In 0.0.5, runtime-image pulls consume the model preparation Job's spec.timeouts.startup budget. On uncached nodes, a slow pull can exhaust the Quick Start's five-minute budget before the preparation process starts; allow more time through that model setting and the CLI's --timeout. Automatic registry selection can also choose a proxy whose manifest is reachable but whose image layers are unavail...

Foretoken 0.0.4

Highlights

• Expanded performance and quality evaluation with foretoken perf and foretoken eval: multi-turn conversations, recorded workloads, arrival patterns, SLO concurrency search, model comparisons, and saved-result plots.
• Multi-node model groups, additional parallelism configurations, and encoder/prefill/decode serving paths, with inference-engine arguments configured through engineArgs.
• Stateless OpenAI Responses and Anthropic Messages APIs, plus additional load- and cache-aware routing algorithms.
• Persistent serving logs, speculative-decoding metrics, Slack and DingTalk alerts, and PyTorch, NVIDIA Nsight Systems, and MetaX mcTracer profiling.
• Local kind/k3d cluster lifecycle commands and incremental source deployments that reuse Kubernetes build caches and unchanged runtime dependencies.

Changes

Distributed serving and model lifecycle

• Multi-node model replicas. A replica can span multiple Kubernetes nodes through LeaderWorkerSet. Its members start, become ready, restart, and drain as one unit, with inference routed through the group leader. (#155)
• Independent Encoder, Prefill, and Decode capacity. E/P/D stages can use different replica counts, resources, and engine parallelism settings. Each stage scales independently while routing selects compatible stage groups. Mooncake transfers now support additional context-, pipeline-, and expert-parallel layouts. Prefill and Decode must use matching PCP/DCP layouts, and heterogeneous tensor-parallel sizes must divide one another. The maintained E/P/D example requires a source-built E/P/D runtime and ReadWriteMany encoder-cache storage. See the E/P/D guide. (#162)
• Explicit TCP or RDMA transport. P/D deployments can select Mooncake TCP, which stages GPU KV data through host memory, or RDMA with platform-managed device allocation. TCP must be configured explicitly; an RDMA deployment does not silently fall back to TCP.
• CPU-side model preparation. Cache-backed vLLM services prepare remote model and tokenizer files in CPU-only Jobs before occupying serving GPUs. Preparation resumes provider downloads, downloads independent artifacts concurrently, and reports progress. Startup and recovery reuse the resolved checkpoint instead of silently selecting a newer remote version. (#195)
• Optional peer-assisted distribution. Dragonfly distributes public model files through node-local peers; authenticated or custom-endpoint downloads use the provider directly. ModelExpress can transfer loaded weights from compatible ready replicas. Both integrations are opt-in. ModelExpress requires a compatible CUDA runtime, allocated RDMA devices, persistent prepared files, DP=1, and no expert-parallel load balancing.
• More reliable updates and cleanup. Frontend readiness follows the requested Deployment template rather than an older available replica. Model deletion recognizes routing withdrawal even when losing the final backend makes a frontend unready. On a fully allocated cluster, an explicitly Unschedulable replacement can trigger draining of the old serving group to free GPUs; see the upgrade notes for availability implications.

Shared KV cache and storage

• Shared-prefix-aware routing. Routing can consider reusable prefixes in a shared Mooncake memory/SSD Store as well as local accelerator caches. A temporarily unavailable shared lookup does not exclude an otherwise eligible backend. (#122)
• Data-parallel cache locality. KV events and shared-prefix observations are tracked per data-parallel rank. A disconnected rank or an event-sequence gap invalidates that rank's locality without discarding healthy peers' observations.
• Memory-only and mixed-tier Stores. A Store can combine memory-only pools with SSD-backed pools. Client disk storage is optional, and multiple models can share capacity while their cache entries remain isolated by model and compatible layout.
• Opt-in Mooncake Master HA. KVService supports two Masters with external etcd, leader discovery, and standby-first updates. HA needs two schedulable nodes and cannot be combined with single-Master snapshots.
• Readable capacity and reusable caches. KV capacities accept Kubernetes quantities such as 1Gi and 1.5Gi. Node-specific Triton caches persist across Pod replacement. Runtime-cache permissions and placement now accommodate preparation, serving, and frontend consumers; non-k3d deployments can turn relative cache directories into dynamically provisioned PVCs, using RWO on a single-node cluster.

Request routing and autoscaling

• Six additional routing scorers: active_request balances frontend-observed in-flight requests; token_load considers in-flight tokens and incoming uncached prompt-token demand; prefix ranks complete cached prompt blocks; no_hit_lru spreads cold requests; load_aware uses waiting-queue thresholds; and two_tier combines active-request imbalance with device-prefix overlap. two_tier requires the max picker. See routing algorithms and configuration.
• Score-aware selection. The default gamble_sampling picker gives higher-ranked targets greater selection probability without excluding the rest. power_of_two_choices and max provide alternative selection strategies.
• AIMD autoscaling. Additive-increase/multiplicative-decrease scaling adds capacity above the queue threshold and reduces it when both waiting and active requests are zero. Recommendations respect the selected adjustment stage and replica bounds. (#90)
• Consistent stage configuration. Autoscaling Trigger, Decision, and Adjustment stages use algorithm with optional parameters. See autoscaling. (#158)

Client APIs and structured generation

• Anthropic Messages and OpenAI Responses. Native Messages, Messages token counting, and stateless Responses join Chat Completions, with streaming and non-streaming output. Clients execute tools and replay conversation history; Responses uses store: false rather than server-stored conversations. See the frontend API guide. (#184)
• Reliable strict tool calls. Model features can declare structuralTag for named, required, and strict tool calls. The vLLM adapter consistently uses Xgrammar so JSON Schema and structural-tag requests can coexist without request-order-dependent grammar failures. (#206)
• Streaming termination fixes. Adapters preserve reasoning, tool output, lifecycle events, and output-budget exhaustion. Failed completion candidates terminate the response rather than leaving other candidate streams hanging, and truncated tool calls remain incomplete.

Performance benchmarking and quality evaluation

• Richer workloads. foretoken perf supports fixed or variable request lengths, local and Hugging Face datasets, multi-turn ShareGPT/StudyChat conversations, tool history, weighted dataset mixes, recorded trace replay, and synthetic prefix-reuse workloads. Constant, Poisson, and gamma arrivals support both steady and bursty traffic.
• Repeatable measurements. Warmup, duration-based runs, repeated measurements, and quieter output simplify experiments. Saved results include workload and environment metadata, replica information, client measurements, and available server observations.
• SLO-aware concurrency search. Search for concurrency under latency objectives, with request-level attainment and goodput, aggregate throughput criteria, observed peak concurrency, and explicit stop reasons. Throughput is evaluated as an aggregate measurement, not a per-request SLO.
• Sweeps and comparisons. JSONL parameter combinations can compare services, deployment configurations, and endpoint URLs. Repeated runs produce error bars and comparison plots. foretoken plot RESULT_DIR redraws saved results in PNG, PDF, or SVG, with CSV exports and publication-oriented layouts; W&B reporting is optional.
• Task-level quality evaluation. foretoken eval integrates lm-eval and EvalScope, supports resumable evaluation, and adds likelihood/perplexity workflows requiring input-token log probabilities. The documented Foretoken scoring path uses a source-built platform; an existing compatible Completions endpoint can also be used.
• Reference/candidate model comparisons. Compare full-vocabulary distributions using KL divergence, total variation, top-1 agreement, top-k overlap, centered-logit RMSE, and token probabilities. Greedy-generation comparison reports exact token-sequence matches and the first divergence; quantized candidates can carry explicit comparison metadata.
• Video benchmarking. foretoken perf video measures synchronous video endpoints with warmup, concurrency, duration, output retention, and sweeps, including VideoArgusBench text/image/video-conditioned workload mappings. Video serving uses the separately configured Omni runtime described below.

See the benchmark guide for workload formats, scoring requirements, and result interpretation.

Observability, logs, and profiling

• Expanded bilingual dashboards. Grafana views cover request/token throughput, q...

Foretoken 0.0.3

Highlights

• Refreshed benchmarks with conversations, recorded tool histories, synthetic output lengths, and W&B time-series and task comparisons.
• Model sources selectable per service: Hugging Face, ModelScope, or local directories, with directory-backed runtime storage.
• Automatic NVIDIA and MetaX runtime selection through foretoken install, with release images for both GPU families.
• LoadBalancer discovery and optional managed address allocation during installation.
• Expanded routing, Mooncake KV lifecycle support, and the Foretoken System Overview dashboard with alerts and diagnostics.

Thanks

• @RSL459 for the llm-d-inspired queue-depth, KV-cache utilization, and running-request routing scorers.
• @YehongXu1 for alert delivery integration and observability runbooks.
• @shiweijiezero for the benchmark refactor, storage and model-source support, platform integration, and release validation.

Install

pip install --upgrade foretoken

For benchmarking:

pip install --upgrade 'foretoken[bench]'

Python 3.11 or later is required. Use the examples from this release and follow the deployment guides for platform upgrades.

Full changelog: v0.0.2...v0.0.3

v0.0.2

What's Changed

• docs(cli): clarify post-release platform reuse by @shiweijiezero in #87
• docs(release): clarify post-release platform artifacts by @shiweijiezero in #88
• feat(router): expose EPD routing progress by @shiweijiezero in #84
• feat: persist model and runtime caches by @shiweijiezero in #89
• docs: hide controller-owned resources from user paths by @shiweijiezero in #95

Full Changelog: v0.0.1a1...v0.0.2

Foretoken 0.0.1 Alpha 1

First public Alpha release of Foretoken.

Install the Python package with:

pip install --pre foretoken

The matching platform artifacts use OCI and Helm version 0.0.1-alpha.1.