Skip to content

The HyperPod Update & Draft Model Support

Choose a tag to compare

@dferguson992 dferguson992 released this 23 Sep 14:16
· 211 commits to main since this release
fdf9084

ML Container Creator v1.7.2 — Release Notes

Overview

v1.7.2 is a quality and completeness patch on top of v1.7.1. It introduces do/draft (speculative decoding management), the draft-model-picker MCP server, deep fixes to --recommend --apply for non-realtime targets, a smarter Bedrock analysis output, and mcc regenerate runtime variable preservation. Several latent bugs were also squashed across HyperPod CLI scripts.


Speculative Decoding Management

do/draft — new command

A dedicated CLI surface for configuring EAGLE3 and other speculative decoding algorithms on HyperPod EKS:

./do/draft list # browse catalog
./do/draft list --target Llama-3.1-8B --limit 10 # filter
./do/draft set thoughtworks/Llama-3.1-8B-Instruct-Eagle3 # configure
./do/draft set thoughtworks/Llama-3.1-8B-Instruct-Eagle3 --deploy # configure + deploy
./do/draft status # show current
./do/draft remove # clear

--deploy runs do/clean hyperpod --force before do/deploy — required because speculative decoding is a vLLM/SGLang startup flag (not hot-loadable) and a clean is needed to avoid GPU resource deadlock on single-node clusters.

Engine guards prevent misconfiguration: vLLM accepts all algorithms; SGLang rejects ngram and medusa; LMI produces a warning with a confirmation prompt; other engines exit with a clear error. s3:// draft model URIs are rejected (HF IDs only, phase 1).

Supported algorithms: eagle3, eagle2, eagle, draft-model, ngram, mtp.

draft-model-picker MCP server

New bundled MCP server backed by servers/lib/catalogs/draft-models.json:

  • list_draft_models — browse with target_model, algorithm, and limit filters
  • get_draft_model — full metadata for a specific HF model ID
  • recommend_draft — best match for a target model, ranked eagle3 → eagle2 → draft-model

Initial catalog: validated EAGLE3 heads for Llama-3.1-8B (thoughtworks + NVIDIA), Llama-3.1-70B, Llama-3.3-70B, and a Vicuna-7B draft head. The catalog (servers/lib/catalogs/draft-models.json) is schema-validated in CI via validate-servers.js and copied to .mlcc/draft-models.json at mcc generate time.


Benchmark Analytics Fixes

--recommend --apply target-aware config key mapping

The DIMENSION_CONFIG_KEY map (dimension → do/config var) was hardcoded to IC-prefixed vars (IC_ENV_VLLM_TENSOR_PARALLEL_SIZE) regardless of deployment target. On HyperPod EKS and async-inference, the correct vars are bare VLLM_* names consumed by the serve wrapper. Fixed:

Target tensor_parallel_degree → quantization →
realtime-inference IC_ENV_VLLM_TENSOR_PARALLEL_SIZE IC_ENV_VLLM_QUANTIZATION
hyperpod-eks VLLM_TENSOR_PARALLEL_SIZE VLLM_QUANTIZATION
async-inference VLLM_TENSOR_PARALLEL_SIZE VLLM_QUANTIZATION
batch-transform N/A — exits with clear message —

On HyperPod EKS, --apply also writes HP_GPU_COUNT alongside VLLM_TENSOR_PARALLEL_SIZE to keep the CRD resource request in sync. The current serving config resolution in _run_athena_recommendations is now target-aware too — it reads VLLM_* vars on HyperPod/async instead of the IC-only IC_ENV_* vars that were never set on those targets.

--compare-baseline human message when no runs exist after pin

--compare-baseline previously returned a raw JSON blob ({"status": "no_baseline", ...}) when a baseline was pinned but no subsequent runs existed. Now prints a clear human message:

📌 Pinned baseline: fast-bert-model-benchmark-20260925-140807
No benchmark runs found after the pinned baseline.
Run a benchmark to generate a comparison:
./do/benchmark --workload sample
./do/benchmark --compare-baseline
To clear the pin and compare against all historical runs:
./do/benchmark --set-baseline --clear

--json mode still returns the structured object for scripting.

--recommend --apply result messaging fix

When --apply found no recommendations (already at optimal config), it printed "✅ Recommendations applied" regardless. Now correctly shows "ℹ️ No changes applied — current config is already optimal" and removes the unnecessary backup files.

Bedrock interpretation — structured output with richer context

bedrock_interpret() now:

  • Returns a structured dict {headline, risks: [...], next_steps: [...]} instead of a freeform string
  • Passes workload evidence (best metric per workload from the Athena records), model architecture (parameter count, MoE/dense, context length from model-sizes.json), instance specs (GPU count/type/VRAM from instances.json), and active speculative decoding config to the prompt
  • System prompt instructs the model to identify interaction risks between settings (e.g. high TP + speculative decoding, NCCL overhead at low concurrency)
  • Token budget increased from 300 → 600 to accommodate structured output

Display renders as:

💡 Analysis:
<headline>
⚠️ Risks:
• <risk 1>
• <risk 2>
✅ Suggested next steps:
• <step 1>

Phase 2 GPU Metrics Fix

--gpu-metrics was always showing metrics_source=none because the engine metrics collection ran only in the single-level benchmark path, not in the multi-level path (which is the default for all named workloads). Fixed:

  • _collect_engine_metrics_file() now also runs in the multi-level result persist block
  • Collected metrics are written to benchmarks/.last_gpu_metrics.json (fixed location) so --gpu-metrics reliably finds them regardless of which benchmark run completed most recently
  • benchmark_gpu_metrics.py infers metrics_source from which keys are populated when the sidecar doesn't include an explicit metrics_source field

mcc regenerate — Runtime Variable Preservation

mcc regenerate previously wiped all runtime-written vars from do/config (HP cluster name, endpoint name, speculative config, tensor parallel size, pinned baseline, etc.), requiring manual re-entry after every regeneration.

Fixed: RUNTIME_OWNED_VARS in regenerate-command-handler.js lists all vars written at runtime by do/deploy, do/draft, do/benchmark, do/optimize --apply, and do/benchmark --set-baseline. Before writeProject runs, these are captured from the existing do/config; after rendering, they are re-injected over any template defaults.

Output: ♻️ Preserved N runtime vars in do/config.

To add a new runtime-written var, append it to RUNTIME_OWNED_VARS — no other changes needed.


HyperPod CLI Profile Bug Fixes

do/logs, do/test, do/clean.d/hyperpod-eks, and do/adapter were calling aws sagemaker describe-cluster and aws eks update-kubeconfig without --profile, causing authentication failures in environments that require an explicit AWS profile. All four scripts now pass --profile ${_PROFILE_awsProfile} when the profile is set. (do/deploy and do/benchmark already had this fix from BL101.)


Terminology Fix

The suggestion printed by ./do/benchmark --recommend when no benchmark history is available referred to "SageMaker AI Recommendations API" — the correct product name is SageMaker AI Inference Optimizer. Fixed in the output message and the --help text.

The suggested next-step command ./do/deploy --optimize (which was never implemented) has been corrected to ./do/deploy.