Inference Autopilot v0.1.25 — Adaptive Search and Confirmation Scheduling

Technical highlights

Cookbook and live-version compatibility

• Extract static and dynamically generated, model-specific recipes from the current SGLang Cookbook, including JSX-based configuration matrices.
• Validate every documented flag and value against the installed SGLang ServerArgs contract before execution.
• Translate deprecated Cookbook flags only when the live CLI explicitly names their replacement.
• Adapt documented TP/PP/DP/EP topology to the selected GPU pool while preserving independent model and workload controls and removing only incompatible distributed companions.
• Treat model-specific recipe handlers as compatibility adapters; candidate selection remains driven by the live model, hardware, workload, SLO, and measured evidence.

Context-aware adaptive search

• Reduce Nsys, runtime-log, model, workload, and memory evidence into canonical bottleneck classes, including co-primary classes when scores are effectively tied.
• Use a unified Candidate Registry to track applicability, dependencies, conflicts, risk, scheduling, measurements, rejection reasons, and parent/child relationships.
• Match parameters to the full (hardware, model, workload, SLO, bottleneck) context instead of relying on one global ranking.
• Cover distinct semantic controls within each mechanism before spending additional budget on nearby values of the same parameter.
• Derive continuous candidate values from effective defaults, VRAM headroom, workload length, topology, and allocator reserves.

Resource-aware execution

• Pack independent candidates onto disjoint GPUs according to each configuration's TP/PP/DP requirement.
• Backfill otherwise idle GPU slots with valuable deferred refinements only when doing so does not add an estimated execution wave or consume confirmation reserves.
• Compare topology candidates with resource-normalized objectives such as throughput per GPU instead of allowing wider configurations to win on raw service throughput alone.
• Keep plan-time and runtime Nsys-overlap qualification consistent and report why overlap was disabled.

Composition and statistical confirmation

• Use champion augmentation to test whether an individually positive parameter still adds value when composed with the current best configuration.
• Require generated combinations to beat their strongest measured direct parent before they can enter confirmation.
• Confirm the first nominee against the original baseline, then compare competitive challengers directly with the confirmed champion.
• Reserve minimum evidence for later challengers so one ambiguous candidate cannot consume the complete confirmation budget.
• Stop early with Bayesian sequential evidence when a gain or loss is already clear.
• Run same-configuration confirmation replicas on disjoint GPUs and reuse resident services across repeated windows when a configuration occupies the full GPU pool.

Profiling and auditability

• Keep fresh and reused Nsys capacity evidence consistent by recording the actual pressure ceiling used by profiling.
• Preserve the original safe baseline even after champion/challenger comparisons.
• Report GPU placement, effective runtime settings, Cookbook adaptation, bottleneck evidence, idle-slot backfill, composition edges, statistical anchors, and replica/resident execution.

Validation

• 235 tests pass on the published repository snapshot.
• Real GPU validation covered Cookbook discovery, resource-normalized topology comparison, idle-GPU refinement backfill, champion augmentation, Bayesian confirmation, and replica-parallel resident confirmation.

Install

python3 -m pip install \
  --no-deps \
  --force-reinstall \
  "git+https://github.com/Jacki1223/inference-autopilot.git@v0.1.25"

See the README for the end-to-end inferopt init -> doctor -> plan -> run -> report workflow.

Inference Autopilot v0.1.24 — Initial Release

Initial release

Inference Autopilot is an evidence-driven CLI for hardware- and workload-aware SGLang deployment optimization. It discovers the installed SGLang parameter surface, validates feasibility, benchmarks applicable configurations, profiles the serving path, confirms gains under SLO and stability gates, and emits reproducible deployment commands.

Highlights

• Canonical Candidate Registry with merged provenance, lifecycle state, risk, dependencies, measurements, and parent/child relationships.
• Mechanism-level adaptive search that covers applicable mechanisms before deep refinement, then spends follow-up budget only on positive or uncertain evidence.
• Workload- and hardware-derived parameter values for prefill chunking, KV capacity, scheduling, CUDA Graph, model-native mechanisms, and topology.
• Nsight Systems and SGLang log analysis with a canonical bottleneck classifier and intervention-informed posterior diagnosis.
• Parent-relative composition gates so weak or conflicting parameter combinations do not pollute the final command.
• Bayesian repeated-window confirmation, SLO gating, confidence intervals, and explicit baseline retention.
• Safe resume semantics: raw profiles may be reused, while search-policy changes invalidate screening and downstream decisions.
• Minimal and reproducible SGLang launch commands, structured JSON artifacts, and a Markdown report.

Validation

• 159 automated tests pass.
• The wheel was built, installed in isolation, and imported successfully.
• End-to-end GPU experiments verified screening, refinement, composition, confirmation, reporting, cleanup, and resume behavior.

Install

python3 -m pip install \
  --no-deps \
  --no-build-isolation \
  --force-reinstall \
  "git+https://github.com/Jacki1223/inference-autopilot.git@v0.1.24"

This is the initial public release. Results are intentionally bounded to the recorded model, SGLang version, hardware, workload, experiment budget, and acceptance gates; they are not claims of a universal optimum.