Foretoken 0.0.4

Highlights

• Expanded performance and quality evaluation with foretoken perf and foretoken eval: multi-turn conversations, recorded workloads, arrival patterns, SLO concurrency search, model comparisons, and saved-result plots.
• Multi-node model groups, additional parallelism configurations, and encoder/prefill/decode serving paths, with inference-engine arguments configured through engineArgs.
• Stateless OpenAI Responses and Anthropic Messages APIs, plus additional load- and cache-aware routing algorithms.
• Persistent serving logs, speculative-decoding metrics, Slack and DingTalk alerts, and PyTorch, NVIDIA Nsight Systems, and MetaX mcTracer profiling.
• Local kind/k3d cluster lifecycle commands and incremental source deployments that reuse Kubernetes build caches and unchanged runtime dependencies.

Changes

Distributed serving and model lifecycle

• Multi-node model replicas. A replica can span multiple Kubernetes nodes through LeaderWorkerSet. Its members start, become ready, restart, and drain as one unit, with inference routed through the group leader. (#155)
• Independent Encoder, Prefill, and Decode capacity. E/P/D stages can use different replica counts, resources, and engine parallelism settings. Each stage scales independently while routing selects compatible stage groups. Mooncake transfers now support additional context-, pipeline-, and expert-parallel layouts. Prefill and Decode must use matching PCP/DCP layouts, and heterogeneous tensor-parallel sizes must divide one another. The maintained E/P/D example requires a source-built E/P/D runtime and ReadWriteMany encoder-cache storage. See the E/P/D guide. (#162)
• Explicit TCP or RDMA transport. P/D deployments can select Mooncake TCP, which stages GPU KV data through host memory, or RDMA with platform-managed device allocation. TCP must be configured explicitly; an RDMA deployment does not silently fall back to TCP.
• CPU-side model preparation. Cache-backed vLLM services prepare remote model and tokenizer files in CPU-only Jobs before occupying serving GPUs. Preparation resumes provider downloads, downloads independent artifacts concurrently, and reports progress. Startup and recovery reuse the resolved checkpoint instead of silently selecting a newer remote version. (#195)
• Optional peer-assisted distribution. Dragonfly distributes public model files through node-local peers; authenticated or custom-endpoint downloads use the provider directly. ModelExpress can transfer loaded weights from compatible ready replicas. Both integrations are opt-in. ModelExpress requires a compatible CUDA runtime, allocated RDMA devices, persistent prepared files, DP=1, and no expert-parallel load balancing.
• More reliable updates and cleanup. Frontend readiness follows the requested Deployment template rather than an older available replica. Model deletion recognizes routing withdrawal even when losing the final backend makes a frontend unready. On a fully allocated cluster, an explicitly Unschedulable replacement can trigger draining of the old serving group to free GPUs; see the upgrade notes for availability implications.

Shared KV cache and storage

• Shared-prefix-aware routing. Routing can consider reusable prefixes in a shared Mooncake memory/SSD Store as well as local accelerator caches. A temporarily unavailable shared lookup does not exclude an otherwise eligible backend. (#122)
• Data-parallel cache locality. KV events and shared-prefix observations are tracked per data-parallel rank. A disconnected rank or an event-sequence gap invalidates that rank's locality without discarding healthy peers' observations.
• Memory-only and mixed-tier Stores. A Store can combine memory-only pools with SSD-backed pools. Client disk storage is optional, and multiple models can share capacity while their cache entries remain isolated by model and compatible layout.
• Opt-in Mooncake Master HA. KVService supports two Masters with external etcd, leader discovery, and standby-first updates. HA needs two schedulable nodes and cannot be combined with single-Master snapshots.
• Readable capacity and reusable caches. KV capacities accept Kubernetes quantities such as 1Gi and 1.5Gi. Node-specific Triton caches persist across Pod replacement. Runtime-cache permissions and placement now accommodate preparation, serving, and frontend consumers; non-k3d deployments can turn relative cache directories into dynamically provisioned PVCs, using RWO on a single-node cluster.

Request routing and autoscaling

• Six additional routing scorers: active_request balances frontend-observed in-flight requests; token_load considers in-flight tokens and incoming uncached prompt-token demand; prefix ranks complete cached prompt blocks; no_hit_lru spreads cold requests; load_aware uses waiting-queue thresholds; and two_tier combines active-request imbalance with device-prefix overlap. two_tier requires the max picker. See routing algorithms and configuration.
• Score-aware selection. The default gamble_sampling picker gives higher-ranked targets greater selection probability without excluding the rest. power_of_two_choices and max provide alternative selection strategies.
• AIMD autoscaling. Additive-increase/multiplicative-decrease scaling adds capacity above the queue threshold and reduces it when both waiting and active requests are zero. Recommendations respect the selected adjustment stage and replica bounds. (#90)
• Consistent stage configuration. Autoscaling Trigger, Decision, and Adjustment stages use algorithm with optional parameters. See autoscaling. (#158)

Client APIs and structured generation

• Anthropic Messages and OpenAI Responses. Native Messages, Messages token counting, and stateless Responses join Chat Completions, with streaming and non-streaming output. Clients execute tools and replay conversation history; Responses uses store: false rather than server-stored conversations. See the frontend API guide. (#184)
• Reliable strict tool calls. Model features can declare structuralTag for named, required, and strict tool calls. The vLLM adapter consistently uses Xgrammar so JSON Schema and structural-tag requests can coexist without request-order-dependent grammar failures. (#206)
• Streaming termination fixes. Adapters preserve reasoning, tool output, lifecycle events, and output-budget exhaustion. Failed completion candidates terminate the response rather than leaving other candidate streams hanging, and truncated tool calls remain incomplete.

Performance benchmarking and quality evaluation

• Richer workloads. foretoken perf supports fixed or variable request lengths, local and Hugging Face datasets, multi-turn ShareGPT/StudyChat conversations, tool history, weighted dataset mixes, recorded trace replay, and synthetic prefix-reuse workloads. Constant, Poisson, and gamma arrivals support both steady and bursty traffic.
• Repeatable measurements. Warmup, duration-based runs, repeated measurements, and quieter output simplify experiments. Saved results include workload and environment metadata, replica information, client measurements, and available server observations.
• SLO-aware concurrency search. Search for concurrency under latency objectives, with request-level attainment and goodput, aggregate throughput criteria, observed peak concurrency, and explicit stop reasons. Throughput is evaluated as an aggregate measurement, not a per-request SLO.
• Sweeps and comparisons. JSONL parameter combinations can compare services, deployment configurations, and endpoint URLs. Repeated runs produce error bars and comparison plots. foretoken plot RESULT_DIR redraws saved results in PNG, PDF, or SVG, with CSV exports and publication-oriented layouts; W&B reporting is optional.
• Task-level quality evaluation. foretoken eval integrates lm-eval and EvalScope, supports resumable evaluation, and adds likelihood/perplexity workflows requiring input-token log probabilities. The documented Foretoken scoring path uses a source-built platform; an existing compatible Completions endpoint can also be used.
• Reference/candidate model comparisons. Compare full-vocabulary distributions using KL divergence, total variation, top-1 agreement, top-k overlap, centered-logit RMSE, and token probabilities. Greedy-generation comparison reports exact token-sequence matches and the first divergence; quantized candidates can carry explicit comparison metadata.
• Video benchmarking. foretoken perf video measures synchronous video endpoints with warmup, concurrency, duration, output retention, and sweeps, including VideoArgusBench text/image/video-conditioned workload mappings. Video serving uses the separately configured Omni runtime described below.

See the benchmark guide for workload formats, scoring requirements, and result interpretation.

Observability, logs, and profiling

• Expanded bilingual dashboards. Grafana views cover request/token throughput, queues, time to first token, end-to-end latency, time per output token, inter-token latency, prefill/decode timing, KV caches, routing, and autoscaling. Namespace, model, instance, role, and rank filters make distributed deployments easier to inspect.
• Speculative-decoding visibility. Metrics include acceptance behavior and draft/target GPU time, alongside the runtime's existing serving measurements.
• Persistent serving logs. Loki-backed logs remain available after model Pods or namespaces are removed while the log store is retained. Defaults provide 14-day retention and 5Gi storage. Optional volume expansion can double capacity at 80% usage up to a configured maximum when the storage driver supports expansion.
• Alert integrations. Services opt into alert rules through spec.observability.alerts.rules, with notification delivery through Slack, Lark, and DingTalk. Grafana can use anonymous access or password authentication; reusing an existing installation preserves its settings.
• Three profiling backends. Deployment and performance workflows support PyTorch, NVIDIA Nsight Systems, and MetaX mcTracer, with capture duration controls and persistent history. foretoken profile view opens the corresponding visualization workflow. Profiling requires a source-installed CLI and platform. Nsight uses a separately built diagnostic image; mcTracer needs the matching MACA SDK tools in the model-server image. For either tool, --profile-engine must match ModelService.spec.profiling.engine. See the profiling setup.

See the observability guide for dashboards, alerts, and profiling setup.

Deployment and source development

• Local cluster lifecycle. foretoken cluster create and foretoken cluster delete manage kind/k3d development clusters, including selected-GPU k3d configurations and persistent data mounts.
• Incremental source iteration. Source deployments synchronize changed files and reuse Kubernetes BuildKit caches and unchanged runtime dependencies. Python/Triton and compiled Rust/CUDA/C++ changes follow their respective rebuild paths, with updated workloads observed through readiness and routing before completion. Separate engine checkouts support inference-engine development without editing vendored reference trees.
• More deployment examples. Tagged examples cover NVIDIA BF16, AWQ, and loading-time BitsAndBytes 4-bit models; independent E/P/D stages; shared KV; MetaX GLM-5.3; and profiling. The source deployment guide explains custom runtime images and iterative development.

Inference runtimes and video serving

• Native inference options. engineArgs accepts native YAML values for context length, precision, quantization, batching, parallelism, and complete speculative-config dictionaries. GPU resources remain specified per member Pod. (#149)
• Reproducible MetaX runtime builds. The runtime uses public SDK packages and a pinned vLLM/core-plugin source pair, including GLM-5.3 attention, sequence-parallel, MTP, and mHC adaptations. A source-deployment recipe covers two-node BF16 serving with a shared memory Store.
• MetaX correctness fixes. Quantized MLA projections retain their native quantization loaders; INT8 MoE kernels filter nonlocal experts correctly. DP>1 configurations with expert parallelism serialize shared-expert execution on the existing main-stream path, without changing other platforms or single-DP execution. (#197, #206)
• Engine compatibility fixes. Updated EngineCore integration handles newer vLLM layouts while retaining the NVIDIA 0.26 runtime. Release fixes address ready-response patch application, development-version profiling selection, and MetaX sequence-length metadata during graph execution.
• Opt-in MiniMax H3 integration. A source-built vLLM-Omni adapter provides synchronous video serving and a two-GPU BF16 H3 recipe with image/video conditioning. Omni currently uses single-node aggregate serving, not Foretoken KV, E/P/D, or profiling configuration. No Omni runtime image is included in the standard release artifacts. (#183)
• Durable asynchronous video tasks. The optional VideoTask API supports submission, status, cancellation, result download, deletion, and expiration through Kubernetes worker Jobs and a dedicated PVC. Tasks survive frontend/controller restarts without silently repeating an uncertain generation. Large synchronous multipart uploads are spooled to disk rather than buffered under the text JSON body limit. (#205)

Upgrade Notes

This release changes several v1alpha1 configuration fields. Back up existing custom resources and migrate their manifests before updating the platform. Use the 0.0.4 examples, rather than mixing examples from a newer main with release images.

• Replace parallelism and extraArgs with engineArgs. For example, parallelism.tp: 2 becomes engineArgs: {tensor-parallel-size: 2}. Pool-level engineArgs replaces the service-level argument map; it is not merged. See inference parameters.

• Express router stages as objects with algorithm and optional parameters. The round_robin picker has been removed; the default is gamble_sampling. See request routing.

• Move queue settings from decision.queue or decision.queueThreshold into decision.parameters; move trigger.interval to trigger.parameters.interval. Rename adjustment.scaleUp.stabilizationWindow and adjustment.scaleDown.stabilizationWindow to adjustment.parameters.scaleUpStabilizationWindow and adjustment.parameters.scaleDownStabilizationWindow. See autoscaling configuration. Updating CRDs does not migrate existing resource fields.

• Replace foretoken bench with foretoken perf; use --max-concurrency, --num-prompts, and --request-rate instead of --parallel, --number, and --rate. Quality evaluation uses foretoken eval. Benchmark dependencies are now included in the normal package installation; the former bench extra is no longer needed.

• Upgrade CRDs, controller, frontend, and model-server together; do not mix 0.0.3 and 0.0.4 components.

• Expert parallelism now uses enable-expert-parallel, data-parallel-size, all2all-backend, and enable-eplb in engineArgs. EP rank count is derived from TP × PCP × DP rather than a separate parallelism.ep.size input.

• Scripts reading E/P/D autoscaling status should select individual Pool targets instead of the former combined EPD target.

• For custom Helm values, move P/D RDMA allocation to top-level rdma.resourceName and rdma.resourceCount. Remove runtime.vllm.pd.profileName/profileRevision; P/D uses runtime.vllm.profileRevision and explicit transport settings.

• Reserve spare resources for uninterrupted replacement. If a replacement is explicitly Unschedulable, the controller may drain the previous serving group, interrupting service while the replacement loads.

• Switching a KVService between single-Master and HA modes loses cached KV state; it is not an in-place failover upgrade.

The former deploy/dev-deploy helper has been removed. Use foretoken cluster create and foretoken cluster delete for local cluster lifecycle operations; see the CLI guide.

For an installation using release images, migrate manifests before updating the CLI and platform:

pip install --upgrade 'foretoken==0.0.4'
foretoken install

Redeploy affected model services with foretoken deploy PATH, where PATH is the migrated Kustomize deployment directory.

For an existing source installation, update the checkout and editable CLI installation, then use foretoken install -e ., retaining the original registry, values, and engine-source options; follow the source deployment guide.

Python 3.11 or later and Kubernetes 1.29 or later are required. OCI images target Linux/amd64. The NVIDIA model-server bundles vLLM 0.26.0, PyTorch 2.11.0, and CUDA 13.0.2; the host driver must support that runtime. MetaX uses the pinned source-built runtime with MACA 3.8.2.6, MetaX PyTorch 2.10.0, and vLLM 0.30.0.dev0.

Release Artifacts

Artifact	Version or tag
Python package	foretoken==0.0.4
Control plane	ghcr.io/shiweijiezero/foretoken/control-plane:0.0.4
Frontend	ghcr.io/shiweijiezero/foretoken/frontend:0.0.4
NVIDIA model-server	ghcr.io/shiweijiezero/foretoken/model-server:0.0.4
MetaX model-server	ghcr.io/shiweijiezero/foretoken/model-server:0.0.4-metax
Helm Chart	oci://ghcr.io/shiweijiezero/foretoken/charts/foretoken, version 0.0.4

The standard artifact set does not include a vLLM-Omni runtime image. MiniMax H3/video serving requires the separate source-build recipe and an explicitly configured Omni image.

Installation Notes

• In the tagged MiniMax H3 recipe, replace picker.algorithm: round_robin with gamble_sampling or max before deployment. The old picker is no longer registered.

• On multi-GPU hosts, k3d nodes exposing only selected GPUs may need the DCGM Exporter Helm release's extraEnv to set NVIDIA_VISIBLE_DEVICES to those GPU UUIDs. Otherwise the NVIDIA runtime may attempt to mount GPUs absent from the node. Retry foretoken install after the exporter is ready.

• A first-time download of a large runtime image can exceed the example's five-minute preparation budget. Increase ModelService.spec.timeouts.startup when needed; the CLI's --timeout does not change that service-side budget.

Release Validation

The published package, Chart, and NVIDIA images were installed in a fresh GPU Kubernetes environment, with model preparation, ordinary and streamed generation, a short performance run with Prometheus observations, and teardown exercised. Candidate validation also covered JSON Schema, strict Chat/Responses tool calls, concurrent JSON/structural-tag requests, and dynamic RWO storage. The MetaX runtime passed single-GPU generation with its default graph execution enabled; this release qualification was not a full multi-node GLM or H3 video-quality evaluation.

Thanks

• @RSL459 for active-request, token-load, prefix, no-hit LRU, load-aware, and two-tier routing scorers.
• @xlpu for video benchmarking, Omni/H3 serving integration, and inference-runtime and GPU-monitoring fixes.
• @YehongXu1 for profiling, repeatable benchmark runs, autoscaling configuration, alert integrations, and quantized-model examples.
• @2GeTi for SLA/SLO concurrency search and search-boundary fixes.
• @xinghuanliukong for the AIMD autoscaling algorithm.
• @shiweijiezero for multi-node and E/P/D integration, native client APIs, evaluation and metrics integration, persistent logs, model distribution, and incremental source deployments.

Full changelog: v0.0.3...v0.0.4