Skip to content

AISimulate Release v0.13.0

Latest

Choose a tag to compare

@saturley-hall saturley-hall released this 21 Oct 23:51
289f8fb

AISimulate - Release 0.13.0

Summary

AISimulate 0.13.0 pairs with Dynamo 1.6.0. This release completes the move from AIConfigurator to AISimulate and adds forward-pass model (FPM) self-service onboarding for vLLM, public accuracy reporting, experimental AgentX trace replay, KV cache offload simulation, and broader model and serving-mode coverage.

pip install aisimulate==0.13.0
  • Python 3.11 to 3.13. Python 3.14 is not yet supported; pip reports a resolution failure without a clear hint.
  • Wheels: Linux x86-64, Linux aarch64, macOS arm64.
  • Validated pairing: AISimulate 0.13.0 with Dynamo 1.6.0.

Key Highlights

FPM self-service onboarding (vLLM)

Onboard a new model on your own GPUs by measuring its forward passes. aisimulate onboard init declares the model, target hardware and worker topology. onboard plan writes the collection plan plus predict and recommend configs. onboard collect-fpm previews the collection, then runs Dynamo's vLLM self-benchmark with --execute --smoke and --execute. onboard validate-collection checks the measured data, and onboard finalize produces the simulation inputs. Further subcommands cover runtime probing, importing saved observations, serving validation and resumable checkpoints (#196, #248). Guide: FPM self-service.

  • Runtime probing ships audited instrumentation for vLLM 0.27.0 only; Dynamo 1.6.0 runs vLLM 0.31.0.
  • Collecting dense models needs Dynamo 1.6.0 GA (it includes ai-dynamo/dynamo#16804), and very large checkpoints may not complete (see Known issues).
  • SGLang and TRT-LLM self-service are on the roadmap.

Public accuracy reporting

Forward-pass and end-to-end accuracy dashboards are published at https://ai-dynamo.org/aisimulate/fpm-accuracy/ and https://ai-dynamo.org/aisimulate/e2e-accuracy/ (#253, #368, #375).

AgentX trace replay (experimental)

Replay agentic coding traces offline for vLLM and SGLang: direct Weka AgentX ingestion, deterministic play outcomes, seeded snapshots and continuous duration profiles (#115, #121, #122, #190, #207, #307). In release testing, a finite AgentX trace completed all 722 requests with 275 MB peak memory.

KV cache offload simulation

AISimulate models three KV cache tiers: G1 (GPU memory), G2 (host memory) and G3 (storage).

  • vLLM G2 offload, local and cluster-shared, including attention DP and prefill/decode disaggregation (#358, #381).
  • vLLM G3 write-through offload for predict (#203). G3 restores through G2 and requires aggregated vLLM replay with attention DP 1; it is not available with prefill/decode disaggregation, AgentX replay or recommend.
  • Hybrid models: Kimi K3 cache geometry is inferred from the model configuration (#293), and the G1 state cache for Mamba/KDA layers can be sized manually (#243).
  • Partial prefix hits via vLLM prefix_match_unit (#314). For Kimi K3, see Known issues.

Better recommendations

  • recommend can size for the fewest GPUs: optimization.target: min_gpus selects the smallest configuration found that meets the SLA at a fixed request rate or concurrency. Request-rate traffic also needs optimization.constraints.min_goodput_rps (#270).
  • Bayesian search no longer shows the memory growth seen in 0.12: a bounded release test (64 trials at parallelism 2) peaked at 0.87 GB.
  • Heterogeneous prefill and decode hardware: set engine.workers.prefill.hardware or engine.workers.decode.hardware in disaggregated predict and recommend (#186, #204).

Clearer output and errors

  • Predictions print their estimator, timing source, data version and KV precision.
  • Clear errors for unsupported models, missing data versions, zero-completion runs and unsupported FP8 compute (for example FP8 on A100). Parallel shapes that do not split the model's attention heads are skipped instead of failing.
  • --overwrite now validates the configuration (YAML, unknown sections and fields, value types, missing stacks or output adapters) before clearing previous outputs. A misspelled model or hardware name still clears them; see Known issues.

Serving modes and models

  • Serving modes, all in predict:
    • Attention-FFN disaggregation (AFD), modeled analytically and configured with engine.mode: afd and engine.afd; also in recommend (#107 to #111).
    • Analytical encode/prefill/decode disaggregation: an encoder pool (engine.workers.encoder) with fixed image inputs, in front of aggregated or disaggregated workers; also in recommend. Qwen3-VL models are supported in this release (#156).
    • Context parallelism: parallelism.prefill_context (SGLang models only) and parallelism.decode_context (vLLM and SGLang, model-dependent), predict only (#240).
    • TRT-LLM disaggregated serving (#77) and attention DP with different sizes per prefill and decode pool (#105).
    • N-gram prompt-lookup speculative decoding via engine.speculation, for vLLM with op-level timing; also in recommend (#227).
  • Models:
    • DeepSeek V4.1 Flash: FPM self-service plans it with the required eager execution, and measured SGLang FPM profiles (H200, B200 and GB200 at TP4; B300 at TP2 and TP4; GB300 at TP2) load through the Python SDK. predict has no op-level path for it yet (see Known issues).
    • Kimi K3 (moonshotai/Kimi-K3), and Qwen3.8-Max (Qwen/Qwen3.8-2.4T-A95B and -FP8) on vLLM and SGLang. Qwen3.8-Max needs backend_version: next; the default data version has no FP8 MoE measurements for it.
    • Vision encoder modeling for Qwen3.5, Kimi K2.5 and Gemma 4 through the image options of the bundled aiconfigurator compatibility command (#84, #86).
  • New measurements: opt-in FP8 FlashInfer TRT-LLM MoE data for Qwen3.8-Max on SGLang 0.5.17 (B200, GB200 and GB300; select with engine.moe_kernel_source: sglang_flashinfer_trtllm_moe), GB200 multi-node communication data, and a DeepEP low-latency decode communication model.

Dynamo 1.6.0 integration

Tested with Dynamo 1.6.0 release candidates: Planner estimator, profiler rapid mode, mocker with AISimulate timing, frontend AISimulate router, and the DynoSim tutorial.

Also in this release

  • Slurm deployment and benchmark bundles: --deployment-target slurm in the bundled aiconfigurator compatibility command (or the aisimulate.generator API) writes a single-node Slurm (Pyxis/Enroot) bundle with a deploy job and an AIPerf benchmark job for vLLM, SGLang and TRT-LLM, aggregated or disaggregated (#502, #503 and earlier). See deployment generation.
  • Output adapter interface: recommend --output NAME runs an installed output adapter that writes extra artifacts or observes candidates live (#334). No adapters ship with AISimulate; see the output adapter ABI.
  • Faster replay: scheduler queue indexes, cheaper SGLang admission, faster prompt hashing, reduced bookkeeping (#348, #356, #386, #397).
  • Agent skills for developing with AISimulate (#322).

Migration and Compatibility Notes

  • The aiconfigurator and aiconfigurator_core Python import namespaces are removed. Import from aisimulate and aisimulate_core instead (for example aiconfigurator_core.sdk becomes aisimulate_core.sdk). The aiconfigurator command remains for compatibility workflows. The Rust crate is renamed from aiconfigurator-core to aisimulate-core; Cargo users may alias it under the old dependency key. Estimator construction takes a single ForwardPassPerfModelConfig. Convert any pickles that encode the old module paths before upgrading. See the migration guide.
  • Host resource supervision. Every predict and recommend run now checks host CPU and memory budgets and runs inside a supervised process tree. A prediction refused by the preflight check exits with status 3 and writes resource-plan.json. A recommend sweep that had to skip resource-limited candidates still writes its results but exits with status 3. Initialization times out after 60 seconds by default; on a busy machine, raise execution.resources.initialization_timeout_seconds. See local resources.
  • Prefill context parallelism now counts toward GPU accounting (#491). ReplayEngineConfig and ReplayRoleConfig gain prefill_context_parallel_size (default 1). Rust code that builds these structs with exhaustive literals must add the field or use Default. GPUs per worker are now attention DP x TP x PCP, and GPU-hours integrate that over worker lifetimes, so configurations that use prefill context parallelism report more GPUs than in 0.12 (for example, PCP8 now reports 8 GPUs per worker). Decode context parallelism adds no GPUs.
  • Prediction output gains fields (additive). JSON output and saved prediction.json gain prediction_provenance (#455) and, when memory diagnostics provide it, kv_cache_precision (#445). Where available they are keyed by worker role (aggregated, or prefill and decode): prediction_provenance.<role> holds estimator, timing_source, data_version, data_identity and data_version_unavailable_reason, and kv_cache_precision.<role> holds dtype and source. When per-role timing is not available (AFD, analytical encoder disaggregation, Dynamo stack runs), provenance reports runner.unavailable_reason instead. --detail memory adds kvcache_quant_mode. No existing keys are renamed or removed. Table output adds KV cache (<role>) and Timing (<role>) lines after the metrics table. Parsers that reject unknown fields, or that parse the full table text, need updating.
  • Predictions with no completed requests now fail (#458). predict still saves the report but exits 1. JSON adds prediction_status: invalid, stop_reason: no_completed_requests and a diagnostic, and latency and throughput values are null instead of 0.

Bug Fixes

  • TRT-LLM P/D now retains KV transfer (#504).
  • SGLang Slurm bootstrap listeners bind to node interfaces (#503), and the Slurm guide covers clusters without GPU GRES (#502).
  • Selected KV precision is preserved through runtime commands (#483); the default backend version resolves for KV cache estimates (#490).
  • DeepSeek V4.1 infers required eager execution (#480); fixes to the vLLM DeepSeek Sparse Attention (DSA) collector and projection precision (#424, #426).
  • SGLang CUDA graph flags are versioned for newer runtimes (#472).
  • Explicit hybrid parallel presets are admitted, and GPU budget failures are explained (#453, #456).
  • FPM onboarding explains that GLM mixed-attention models are not yet supported (#481).
  • The Parquet upgrade removes the transitive Thrift dependency (#414).

Known Issues

Issue Who is affected Workaround
FPM self-service cannot complete a full collection for dense models (for example Qwen3) on Dynamo 1.6.0 builds without the FPM engine-provenance fix Self-service users onboarding dense models Fixed in Dynamo 1.6.0 GA (ai-dynamo/dynamo#16804). Dynamo 1.6.0 release candidates before the fix are affected
Collector pods get a fixed 64 GiB of host memory per GPU, and engine logs are written only after a collection cell ends. A 464 GB checkpoint (GLM-5.3 NVFP4, TEP4) was killed (exit 137) or stalled during load with no logs Self-service users onboarding very large checkpoints No workaround in 0.13.0
MiniMax TP4 example collection fails on Dynamo 1.6.0 with a KV warm-up protocol mismatch FPM self-service, MiniMax TP4 example No workaround in 0.13.0
FPM collection fails when the model cache (HF_HOME) is mounted read-only: FlashInfer cubins are written under HF_HOME and the KV warm-up dataset next to it (HF_HOME/../fpm_datasets) Self-service on clusters with read-only shared caches Mount the model cache read-write. Fixed on main; not in 0.13.0
--dynamo-version with 1.4.x, 1.5.x or 1.6.0 fails at execute (the version table stops at 1.3.0), in self-service and in deployment generation Users pairing with Dynamo 1.4 or later Omit --dynamo-version; pass the container image explicitly if needed
--overwrite with a misspelled model or hardware name deletes the previous results before the run fails Users rerunning into an existing output directory Write to a new --output-dir, or copy the previous results first
SGLang predictions run much higher than vLLM and TRT-LLM for the same configuration (for example Qwen3-32B FP8 on H200 at concurrency 256: TTFT 8.7 s on SGLang vs 2.4 s on vLLM), and SGLang sizing accuracy is unresolved SGLang sizing Validate against measurements on your target SGLang deployment before capacity planning. Predictions for other backends are not a substitute
Kimi K2.5 time per output token at high concurrency is overpredicted by up to about 1.9x on B200 with vLLM Kimi K2.5 users sizing for high concurrency Validate high-concurrency workloads against serving measurements
DeepSeek V4.1 Flash has no op-level predict path on any GPU, and recommend stops at the first attention-DP shape DeepSeek V4.1 Flash users Use FPM self-service, or the published SGLang FPM profiles through the Python SDK
The GB300 next data version overpredicts DeepSeek V4 Pro decode by about 3.6x Users selecting backend_version: next on GB300 Use current
g3_offload works in predict only; in recommend it fails with a schema error G3 offload users Use predict
Stopping the CLI with SIGTERM (for example from timeout, Kubernetes or Slurm) leaves a worker process running that can write results later Users running AISimulate under a job scheduler or timeout Stop it with Ctrl-C (SIGINT), which stops the worker. After a SIGTERM, end the leftover aisimulate.resource_worker process
For Kimi K3, hybrid-state prefix reuse turns on or off depending on whether prefix_match_unit is set, shifting TTFT by about 25% Kimi K3 replay and prediction Keep state_cache and prefix_match_unit identical across runs you compare, and treat Kimi K3 prefix-reuse gains as unconfirmed until 0.14

Full Changelog

Full Changelog: v0.12.1...v0.13.0