Repository navigation
AISimulate - Release 0.13.0
Summary
AISimulate 0.13.0 pairs with Dynamo 1.6.0. This release completes the move from AIConfigurator to AISimulate and adds forward-pass model (FPM) self-service onboarding for vLLM, public accuracy reporting, experimental AgentX trace replay, KV cache offload simulation, and broader model and serving-mode coverage.
pip install aisimulate==0.13.0- Python 3.11 to 3.13. Python 3.14 is not yet supported; pip reports a resolution failure without a clear hint.
- Wheels: Linux x86-64, Linux aarch64, macOS arm64.
- Validated pairing: AISimulate 0.13.0 with Dynamo 1.6.0.
Key Highlights
FPM self-service onboarding (vLLM)
Onboard a new model on your own GPUs by measuring its forward passes. aisimulate onboard init declares the model, target hardware and worker topology. onboard plan writes the collection plan plus predict and recommend configs. onboard collect-fpm previews the collection, then runs Dynamo's vLLM self-benchmark with --execute --smoke and --execute. onboard validate-collection checks the measured data, and onboard finalize produces the simulation inputs. Further subcommands cover runtime probing, importing saved observations, serving validation and resumable checkpoints (#196, #248). Guide: FPM self-service.
- Runtime probing ships audited instrumentation for vLLM 0.27.0 only; Dynamo 1.6.0 runs vLLM 0.31.0.
- Collecting dense models needs Dynamo 1.6.0 GA (it includes ai-dynamo/dynamo#16804), and very large checkpoints may not complete (see Known issues).
- SGLang and TRT-LLM self-service are on the roadmap.
Public accuracy reporting
Forward-pass and end-to-end accuracy dashboards are published at https://ai-dynamo.org/aisimulate/fpm-accuracy/ and https://ai-dynamo.org/aisimulate/e2e-accuracy/ (#253, #368, #375).
AgentX trace replay (experimental)
Replay agentic coding traces offline for vLLM and SGLang: direct Weka AgentX ingestion, deterministic play outcomes, seeded snapshots and continuous duration profiles (#115, #121, #122, #190, #207, #307). In release testing, a finite AgentX trace completed all 722 requests with 275 MB peak memory.
KV cache offload simulation
AISimulate models three KV cache tiers: G1 (GPU memory), G2 (host memory) and G3 (storage).
- vLLM G2 offload, local and cluster-shared, including attention DP and prefill/decode disaggregation (#358, #381).
- vLLM G3 write-through offload for
predict(#203). G3 restores through G2 and requires aggregated vLLM replay with attention DP 1; it is not available with prefill/decode disaggregation, AgentX replay orrecommend. - Hybrid models: Kimi K3 cache geometry is inferred from the model configuration (#293), and the G1 state cache for Mamba/KDA layers can be sized manually (#243).
- Partial prefix hits via vLLM
prefix_match_unit(#314). For Kimi K3, see Known issues.
Better recommendations
recommendcan size for the fewest GPUs:optimization.target: min_gpusselects the smallest configuration found that meets the SLA at a fixed request rate or concurrency. Request-rate traffic also needsoptimization.constraints.min_goodput_rps(#270).- Bayesian search no longer shows the memory growth seen in 0.12: a bounded release test (64 trials at parallelism 2) peaked at 0.87 GB.
- Heterogeneous prefill and decode hardware: set
engine.workers.prefill.hardwareorengine.workers.decode.hardwarein disaggregatedpredictandrecommend(#186, #204).
Clearer output and errors
- Predictions print their estimator, timing source, data version and KV precision.
- Clear errors for unsupported models, missing data versions, zero-completion runs and unsupported FP8 compute (for example FP8 on A100). Parallel shapes that do not split the model's attention heads are skipped instead of failing.
--overwritenow validates the configuration (YAML, unknown sections and fields, value types, missing stacks or output adapters) before clearing previous outputs. A misspelled model or hardware name still clears them; see Known issues.
Serving modes and models
- Serving modes, all in
predict:- Attention-FFN disaggregation (AFD), modeled analytically and configured with
engine.mode: afdandengine.afd; also inrecommend(#107 to #111). - Analytical encode/prefill/decode disaggregation: an encoder pool (
engine.workers.encoder) with fixed image inputs, in front of aggregated or disaggregated workers; also inrecommend. Qwen3-VL models are supported in this release (#156). - Context parallelism:
parallelism.prefill_context(SGLang models only) andparallelism.decode_context(vLLM and SGLang, model-dependent),predictonly (#240). - TRT-LLM disaggregated serving (#77) and attention DP with different sizes per prefill and decode pool (#105).
- N-gram prompt-lookup speculative decoding via
engine.speculation, for vLLM with op-level timing; also inrecommend(#227).
- Attention-FFN disaggregation (AFD), modeled analytically and configured with
- Models:
- DeepSeek V4.1 Flash: FPM self-service plans it with the required eager execution, and measured SGLang FPM profiles (H200, B200 and GB200 at TP4; B300 at TP2 and TP4; GB300 at TP2) load through the Python SDK.
predicthas no op-level path for it yet (see Known issues). - Kimi K3 (
moonshotai/Kimi-K3), and Qwen3.8-Max (Qwen/Qwen3.8-2.4T-A95Band-FP8) on vLLM and SGLang. Qwen3.8-Max needsbackend_version: next; the default data version has no FP8 MoE measurements for it. - Vision encoder modeling for Qwen3.5, Kimi K2.5 and Gemma 4 through the image options of the bundled
aiconfiguratorcompatibility command (#84, #86).
- DeepSeek V4.1 Flash: FPM self-service plans it with the required eager execution, and measured SGLang FPM profiles (H200, B200 and GB200 at TP4; B300 at TP2 and TP4; GB300 at TP2) load through the Python SDK.
- New measurements: opt-in FP8 FlashInfer TRT-LLM MoE data for Qwen3.8-Max on SGLang 0.5.17 (B200, GB200 and GB300; select with
engine.moe_kernel_source: sglang_flashinfer_trtllm_moe), GB200 multi-node communication data, and a DeepEP low-latency decode communication model.
Dynamo 1.6.0 integration
Tested with Dynamo 1.6.0 release candidates: Planner estimator, profiler rapid mode, mocker with AISimulate timing, frontend AISimulate router, and the DynoSim tutorial.
Also in this release
- Slurm deployment and benchmark bundles:
--deployment-target slurmin the bundledaiconfiguratorcompatibility command (or theaisimulate.generatorAPI) writes a single-node Slurm (Pyxis/Enroot) bundle with a deploy job and an AIPerf benchmark job for vLLM, SGLang and TRT-LLM, aggregated or disaggregated (#502, #503 and earlier). See deployment generation. - Output adapter interface:
recommend --output NAMEruns an installed output adapter that writes extra artifacts or observes candidates live (#334). No adapters ship with AISimulate; see the output adapter ABI. - Faster replay: scheduler queue indexes, cheaper SGLang admission, faster prompt hashing, reduced bookkeeping (#348, #356, #386, #397).
- Agent skills for developing with AISimulate (#322).
Migration and Compatibility Notes
- The
aiconfiguratorandaiconfigurator_corePython import namespaces are removed. Import fromaisimulateandaisimulate_coreinstead (for exampleaiconfigurator_core.sdkbecomesaisimulate_core.sdk). Theaiconfiguratorcommand remains for compatibility workflows. The Rust crate is renamed fromaiconfigurator-coretoaisimulate-core; Cargo users may alias it under the old dependency key. Estimator construction takes a singleForwardPassPerfModelConfig. Convert any pickles that encode the old module paths before upgrading. See the migration guide. - Host resource supervision. Every
predictandrecommendrun now checks host CPU and memory budgets and runs inside a supervised process tree. A prediction refused by the preflight check exits with status 3 and writesresource-plan.json. Arecommendsweep that had to skip resource-limited candidates still writes its results but exits with status 3. Initialization times out after 60 seconds by default; on a busy machine, raiseexecution.resources.initialization_timeout_seconds. See local resources. - Prefill context parallelism now counts toward GPU accounting (#491).
ReplayEngineConfigandReplayRoleConfiggainprefill_context_parallel_size(default 1). Rust code that builds these structs with exhaustive literals must add the field or useDefault. GPUs per worker are now attention DP x TP x PCP, and GPU-hours integrate that over worker lifetimes, so configurations that use prefill context parallelism report more GPUs than in 0.12 (for example, PCP8 now reports 8 GPUs per worker). Decode context parallelism adds no GPUs. - Prediction output gains fields (additive). JSON output and saved
prediction.jsongainprediction_provenance(#455) and, when memory diagnostics provide it,kv_cache_precision(#445). Where available they are keyed by worker role (aggregated, orprefillanddecode):prediction_provenance.<role>holdsestimator,timing_source,data_version,data_identityanddata_version_unavailable_reason, andkv_cache_precision.<role>holdsdtypeandsource. When per-role timing is not available (AFD, analytical encoder disaggregation, Dynamo stack runs), provenance reportsrunner.unavailable_reasoninstead.--detail memoryaddskvcache_quant_mode. No existing keys are renamed or removed. Table output addsKV cache (<role>)andTiming (<role>)lines after the metrics table. Parsers that reject unknown fields, or that parse the full table text, need updating. - Predictions with no completed requests now fail (#458).
predictstill saves the report but exits 1. JSON addsprediction_status: invalid,stop_reason: no_completed_requestsand adiagnostic, and latency and throughput values arenullinstead of 0.
Bug Fixes
- TRT-LLM P/D now retains KV transfer (#504).
- SGLang Slurm bootstrap listeners bind to node interfaces (#503), and the Slurm guide covers clusters without GPU GRES (#502).
- Selected KV precision is preserved through runtime commands (#483); the default backend version resolves for KV cache estimates (#490).
- DeepSeek V4.1 infers required eager execution (#480); fixes to the vLLM DeepSeek Sparse Attention (DSA) collector and projection precision (#424, #426).
- SGLang CUDA graph flags are versioned for newer runtimes (#472).
- Explicit hybrid parallel presets are admitted, and GPU budget failures are explained (#453, #456).
- FPM onboarding explains that GLM mixed-attention models are not yet supported (#481).
- The Parquet upgrade removes the transitive Thrift dependency (#414).
Known Issues
| Issue | Who is affected | Workaround |
|---|---|---|
| FPM self-service cannot complete a full collection for dense models (for example Qwen3) on Dynamo 1.6.0 builds without the FPM engine-provenance fix | Self-service users onboarding dense models | Fixed in Dynamo 1.6.0 GA (ai-dynamo/dynamo#16804). Dynamo 1.6.0 release candidates before the fix are affected |
| Collector pods get a fixed 64 GiB of host memory per GPU, and engine logs are written only after a collection cell ends. A 464 GB checkpoint (GLM-5.3 NVFP4, TEP4) was killed (exit 137) or stalled during load with no logs | Self-service users onboarding very large checkpoints | No workaround in 0.13.0 |
| MiniMax TP4 example collection fails on Dynamo 1.6.0 with a KV warm-up protocol mismatch | FPM self-service, MiniMax TP4 example | No workaround in 0.13.0 |
FPM collection fails when the model cache (HF_HOME) is mounted read-only: FlashInfer cubins are written under HF_HOME and the KV warm-up dataset next to it (HF_HOME/../fpm_datasets) |
Self-service on clusters with read-only shared caches | Mount the model cache read-write. Fixed on main; not in 0.13.0 |
--dynamo-version with 1.4.x, 1.5.x or 1.6.0 fails at execute (the version table stops at 1.3.0), in self-service and in deployment generation |
Users pairing with Dynamo 1.4 or later | Omit --dynamo-version; pass the container image explicitly if needed |
--overwrite with a misspelled model or hardware name deletes the previous results before the run fails |
Users rerunning into an existing output directory | Write to a new --output-dir, or copy the previous results first |
| SGLang predictions run much higher than vLLM and TRT-LLM for the same configuration (for example Qwen3-32B FP8 on H200 at concurrency 256: TTFT 8.7 s on SGLang vs 2.4 s on vLLM), and SGLang sizing accuracy is unresolved | SGLang sizing | Validate against measurements on your target SGLang deployment before capacity planning. Predictions for other backends are not a substitute |
| Kimi K2.5 time per output token at high concurrency is overpredicted by up to about 1.9x on B200 with vLLM | Kimi K2.5 users sizing for high concurrency | Validate high-concurrency workloads against serving measurements |
DeepSeek V4.1 Flash has no op-level predict path on any GPU, and recommend stops at the first attention-DP shape |
DeepSeek V4.1 Flash users | Use FPM self-service, or the published SGLang FPM profiles through the Python SDK |
The GB300 next data version overpredicts DeepSeek V4 Pro decode by about 3.6x |
Users selecting backend_version: next on GB300 |
Use current |
g3_offload works in predict only; in recommend it fails with a schema error |
G3 offload users | Use predict |
Stopping the CLI with SIGTERM (for example from timeout, Kubernetes or Slurm) leaves a worker process running that can write results later |
Users running AISimulate under a job scheduler or timeout | Stop it with Ctrl-C (SIGINT), which stops the worker. After a SIGTERM, end the leftover aisimulate.resource_worker process |
For Kimi K3, hybrid-state prefix reuse turns on or off depending on whether prefix_match_unit is set, shifting TTFT by about 25% |
Kimi K3 replay and prediction | Keep state_cache and prefix_match_unit identical across runs you compare, and treat Kimi K3 prefix-reuse gains as unconfirmed until 0.14 |
Full Changelog
Full Changelog: v0.12.1...v0.13.0