Repository navigation
The HyperPod Update & Draft Model Support
ML Container Creator v1.7.2 — Release Notes
Overview
v1.7.2 is a quality and completeness patch on top of v1.7.1. It introduces do/draft (speculative decoding management), the draft-model-picker MCP server, deep fixes to --recommend --apply for non-realtime targets, a smarter Bedrock analysis output, and mcc regenerate runtime variable preservation. Several latent bugs were also squashed across HyperPod CLI scripts.
Speculative Decoding Management
do/draft — new command
A dedicated CLI surface for configuring EAGLE3 and other speculative decoding algorithms on HyperPod EKS:
./do/draft list # browse catalog./do/draft list --target Llama-3.1-8B --limit 10 # filter./do/draft set thoughtworks/Llama-3.1-8B-Instruct-Eagle3 # configure./do/draft set thoughtworks/Llama-3.1-8B-Instruct-Eagle3 --deploy # configure + deploy./do/draft status # show current./do/draft remove # clear
--deploy runs do/clean hyperpod --force before do/deploy — required because speculative decoding is a vLLM/SGLang startup flag (not hot-loadable) and a clean is needed to avoid GPU resource deadlock on single-node clusters.
Engine guards prevent misconfiguration: vLLM accepts all algorithms; SGLang rejects ngram and medusa; LMI produces a warning with a confirmation prompt; other engines exit with a clear error. s3:// draft model URIs are rejected (HF IDs only, phase 1).
Supported algorithms: eagle3, eagle2, eagle, draft-model, ngram, mtp.
draft-model-picker MCP server
New bundled MCP server backed by servers/lib/catalogs/draft-models.json:
list_draft_models— browse withtarget_model,algorithm, andlimitfiltersget_draft_model— full metadata for a specific HF model IDrecommend_draft— best match for a target model, ranked eagle3 → eagle2 → draft-model
Initial catalog: validated EAGLE3 heads for Llama-3.1-8B (thoughtworks + NVIDIA), Llama-3.1-70B, Llama-3.3-70B, and a Vicuna-7B draft head. The catalog (servers/lib/catalogs/draft-models.json) is schema-validated in CI via validate-servers.js and copied to .mlcc/draft-models.json at mcc generate time.
Benchmark Analytics Fixes
--recommend --apply target-aware config key mapping
The DIMENSION_CONFIG_KEY map (dimension → do/config var) was hardcoded to IC-prefixed vars (IC_ENV_VLLM_TENSOR_PARALLEL_SIZE) regardless of deployment target. On HyperPod EKS and async-inference, the correct vars are bare VLLM_* names consumed by the serve wrapper. Fixed:
| Target | tensor_parallel_degree → | quantization → |
|---|---|---|
| realtime-inference | IC_ENV_VLLM_TENSOR_PARALLEL_SIZE | IC_ENV_VLLM_QUANTIZATION |
| hyperpod-eks | VLLM_TENSOR_PARALLEL_SIZE | VLLM_QUANTIZATION |
| async-inference | VLLM_TENSOR_PARALLEL_SIZE | VLLM_QUANTIZATION |
| batch-transform | N/A — exits with clear message | — |
On HyperPod EKS, --apply also writes HP_GPU_COUNT alongside VLLM_TENSOR_PARALLEL_SIZE to keep the CRD resource request in sync. The current serving config resolution in _run_athena_recommendations is now target-aware too — it reads VLLM_* vars on HyperPod/async instead of the IC-only IC_ENV_* vars that were never set on those targets.
--compare-baseline human message when no runs exist after pin
--compare-baseline previously returned a raw JSON blob ({"status": "no_baseline", ...}) when a baseline was pinned but no subsequent runs existed. Now prints a clear human message:
📌 Pinned baseline: fast-bert-model-benchmark-20260925-140807No benchmark runs found after the pinned baseline.Run a benchmark to generate a comparison:./do/benchmark --workload sample./do/benchmark --compare-baselineTo clear the pin and compare against all historical runs:./do/benchmark --set-baseline --clear
--json mode still returns the structured object for scripting.
--recommend --apply result messaging fix
When --apply found no recommendations (already at optimal config), it printed "✅ Recommendations applied" regardless. Now correctly shows "ℹ️ No changes applied — current config is already optimal" and removes the unnecessary backup files.
Bedrock interpretation — structured output with richer context
bedrock_interpret() now:
- Returns a structured dict
{headline, risks: [...], next_steps: [...]}instead of a freeform string - Passes workload evidence (best metric per workload from the Athena records), model architecture (parameter count, MoE/dense, context length from
model-sizes.json), instance specs (GPU count/type/VRAM frominstances.json), and active speculative decoding config to the prompt - System prompt instructs the model to identify interaction risks between settings (e.g. high TP + speculative decoding, NCCL overhead at low concurrency)
- Token budget increased from 300 → 600 to accommodate structured output
Display renders as:
💡 Analysis:<headline>⚠️ Risks:• <risk 1>• <risk 2>✅ Suggested next steps:• <step 1>
Phase 2 GPU Metrics Fix
--gpu-metrics was always showing metrics_source=none because the engine metrics collection ran only in the single-level benchmark path, not in the multi-level path (which is the default for all named workloads). Fixed:
_collect_engine_metrics_file()now also runs in the multi-level result persist block- Collected metrics are written to
benchmarks/.last_gpu_metrics.json(fixed location) so--gpu-metricsreliably finds them regardless of which benchmark run completed most recently benchmark_gpu_metrics.pyinfersmetrics_sourcefrom which keys are populated when the sidecar doesn't include an explicitmetrics_sourcefield
mcc regenerate — Runtime Variable Preservation
mcc regenerate previously wiped all runtime-written vars from do/config (HP cluster name, endpoint name, speculative config, tensor parallel size, pinned baseline, etc.), requiring manual re-entry after every regeneration.
Fixed: RUNTIME_OWNED_VARS in regenerate-command-handler.js lists all vars written at runtime by do/deploy, do/draft, do/benchmark, do/optimize --apply, and do/benchmark --set-baseline. Before writeProject runs, these are captured from the existing do/config; after rendering, they are re-injected over any template defaults.
Output: ♻️ Preserved N runtime vars in do/config.
To add a new runtime-written var, append it to RUNTIME_OWNED_VARS — no other changes needed.
HyperPod CLI Profile Bug Fixes
do/logs, do/test, do/clean.d/hyperpod-eks, and do/adapter were calling aws sagemaker describe-cluster and aws eks update-kubeconfig without --profile, causing authentication failures in environments that require an explicit AWS profile. All four scripts now pass --profile ${_PROFILE_awsProfile} when the profile is set. (do/deploy and do/benchmark already had this fix from BL101.)
Terminology Fix
The suggestion printed by ./do/benchmark --recommend when no benchmark history is available referred to "SageMaker AI Recommendations API" — the correct product name is SageMaker AI Inference Optimizer. Fixed in the output message and the --help text.
The suggested next-step command ./do/deploy --optimize (which was never implemented) has been corrected to ./do/deploy.