Hyperloom v1.1.2

Hyperloom v1.1.2

This is a patch release on top of 1.1.1. The session record resumes across the upgrade: ensure_schema migrates a 1.1.1 coordinator.db on the way in.

ROCm 10 becomes a validated stack, bring-up gets a phase of its own, and the automatic supervision layer is retired in favour of explicit resume and recovery. ATOM is supported end to end, KernelForge gains an assembly backend and stops losing work a killed campaign had already proven, and the Session Breakdown now says which axis a session was graded on.

The command line, the environment contract, the installation prerequisites and several report fields do move. The ones that need a plan before upgrading are listed under Breaking changes. CHANGELOG.md is retired (#1589); the full per-change list now lives in docs/release-notes.md.

Highlights

1. ROCm 10 is a validated stack

Docker runs SGLang 0.5.20 and vLLM on ROCm 10 user space, bare-metal vLLM builds from source on a ROCm 10 host, and rocprof-compute is installed and found on TheRock's pip-packaged ROCm (#1549, #1593, #1605). A prebuilt SGLang image whose torch fails under rocprofv3 is marked profiling-unavailable instead of hanging the round. SGLang >= 0.5.18 profiles through TraceLens kernel_shape_tool rather than git-applied roofline patches (#1486, #1593). The six-leg ROCm 10 run also surfaced scheduler stalls that are fixed here: KERNEL entry hooks no longer block the coordinator tick, conc-sweep boot rounds are no longer counted as measured variants, and a trace with zero GPU events stops repeat roofline scheduling with a recorded reason (#1605).

2. ENABLEMENT is the sixth phase of the loop

Bring-up leaves FRAMEWORK_AGENT and sits between PRELUDE and FRAMEWORK_AGENT with its own entry and exit, phase_history rows and report section (#1491). It gets no budget share: it is bounded by the wall clock and ENABLEMENT_MAX_ATTEMPTS, and runnability is decided from the measurement rather than a log scan. A KEEP now ships an ordered replay recipe (setup, build, patch) with a sufficient / insufficient verdict under enablement.recipe in session_breakdown.json (#1414), and a KB recipe carrying code overlays can be replayed into a pip-installed framework (#1563).

3. Automatic supervision is retired; limits and recovery are explicit

The Robustness agent, the runtime recover action and the Monitor/Supervisor auto-resume are removed (#1547). Each benchmark spawn has a hard deadline (INFERENCE_OPTIMIZER_BENCHMARK_TIMEOUT_SEC, default 7800) plus a silence limit (INFERENCE_OPTIMIZER_BENCHMARK_SILENCE_TIMEOUT_SEC, default 600) armed only once a ready marker is seen. Resume refuses ownership it cannot verify instead of guessing, and recover-session --confirm-stopped releases a stopped task's residue under the session lock. Inline MCP actions no longer block the coordinator's event loop (#1577).

4. ATOM is supported end to end

Its recovery workers are reaped, --framework atom defaults the kernel backend to forge, and it is published as supported (#1416). Detailed profiling adds --mark-trace and ATOM's annotation switches only when the installed build supports them (#1509).

5. KernelForge gains an assembly backend and keeps more of its work

forge-loop --kernel-backend assembly optimizes compiler-emitted AMDGPU assembly behind a numerical contract (#1426). Warm starts fall back to a fuzzy KB match across framework versions and GPUs (#1512). A lane's patch is merged against the KEEPs that landed before it instead of being dropped on stale context (#1528), Controller patches integrate in priority order with clean patch hygiene (#1583), and a Controller KEEP reaches the stack ledger in every benchmark mode (#1522). A killed campaign's proven fusions are salvaged instead of reported as REVERT (#1533, #1536), its leftovers no longer fail the next task (#1564), and a stalled authoring transport is retried once (#1534). The rewrite pipeline's caller now owns its task shape and stages (#1596).

6. One attempt ledger replaces the Explore phase

The configuration arm gives up after five benched rounds without a win rather than five variants, a grid that kept an earlier variant is no longer read as dry, and a resumed session extends tested / rejected / name_index instead of replacing them (#1587). A benchmark that served its whole protocol is kept despite a non-zero wrapper exit (#1584).

7. SBD V6 says which axis a session graded on

metadata.grading, per-figure perf and graded_on make AgentX and synthetic sessions distinguishable, and an AgentX comparison missing the axis pair fails closed instead of promoting on output throughput (#1527). The timeline records the events it used to drop (#1585), and every timeline failure has one shape, closing an event cut short by a recording error as interrupted (#1578).

8. Kernel tuning, KB identity and agent selection

GEMM and fusion E2E KEEP thresholds default to 1% instead of 3%, and MoE tuning persists valid candidate tables (#1580). One kernel's KB records share one address however its source and framework version spell it (#1529), and a partitioned card is a different machine in the KB key (#1542). Every agent CLI choice goes through the shared preferred_agent_backend rule on official key names only (#1541), with one definition of the default model ids (#1555). On GPUs without a Hyperloom achievable-peak entry, roofline uses AMD vendor theoretical peaks (#1602), and a CUDA-graph capture failure is classified rather than retried in eager mode (#1526).

Breaking changes

An operator upgrading from 1.1.1 has to plan for these. The removed options are refused loudly; the environment-variable changes are silent.

• Two optimizer options are removed: --conc-sweep-timeout-sec (#1547) and --recipe-kb-strict-fingerprint (#1542). The parser is strict, so a launch or resume command that still carries either exits with unrecognized arguments before the session starts. Benchmark spawns are bounded by the two timeouts in Highlight 3; partition mode and ep are now part of the KB key, so the mismatch the fingerprint flag promised to catch cannot occur.
• Nothing restarts a session automatically any more. The robustness-agent console script and tools/robustness_monitor.sh.example are gone (#1547); resume explicitly with --resume-from. Resume now refuses, before any write or dispatch, when the session holds execution ownership it cannot verify. After confirming the task's process tree, remote workers and Ray actor have stopped, release it with inference_optimizer recover-session --session-dir PATH --confirm-stopped TASK_ID --confirmation-reason TEXT.
• Environment variables are removed or renamed, and nothing refuses them. A box that still exports one starts normally and behaves as though it were unset, so find them by reading launch scripts and .env.
  • LLM_GATEWAY_KEY as the only OpenAI-side credential: rename it to OPENAI_API_KEY (#1541). LLM_API_KEY, AMD_LLM_API_KEY, LLM_GATEWAY_KEY, AMD_API_KEY, LLM_PROXY_BASE_URL, LLM_PROXY_API_KEY and GEAK_WORK_DIR are no longer forwarded to workers.
  • HYPERLOOM_SUPERVISOR, HYPERLOOM_SUPERVISOR_ENFORCE, HYPERLOOM_SUPERVISOR_TICK_STALL_SEC, ROBUSTNESS_AGENT_ROOT, ROBUSTNESS_LLM_RCA_DISABLED: delete them (#1547).
  • INFERENCE_OPTIMIZER_DISABLE_LOCAL_INFERENCEX, INFERENCE_OPTIMIZER_LOCAL_INFERENCEX_ROOT: delete them; baseline benchmarks run from $INFERENCEX_PATH as set (#1456).
  • MAGPIE_PATCH_STRICT: delete it; it had no effect once the atomic-write patch was dropped (#1592).
• Installation prerequisites move with the stack. TraceLens and its dependencies must be pre-installed in the coordinator's Python interpreter; analysis no longer runs pip install -e . per request (#1580). Bare-metal vLLM accepts ROCm 7.2.x (the 0.29.0+rocm723 wheel) or ROCm 10 (a source build of VLLM_SOURCE_REF); any other ROCm stack is rejected, and VLLM_INSTALL_METHOD can only confirm the detected route (#1605). The Slurm sbatch no longer forwards ROCR_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES or CUDA_VISIBLE_DEVICES into the container (#1582).
• Report fields change with no alias kept.
  • SBD V6 conc_sweep comparison rows: baseline_throughput → baseline_value, optimized_throughput → optimized_value (#1527). V6 failure rows are keyed by stage instead of phase (#1578).
  • reports/kernel_optimization_summary.json moves to schema_version 2 with a lane_totals block, and kernel_opt_outcome can be unvalidated (#1605).
  • A concurrency sweep that produced no comparable pair stops as sweep_failed rather than sweep_done (#1605).
• Defaults that changed: plateau_overrides.explore_empty_streak counts benched rounds rather than variants (#1587); the GEMM and fusion E2E KEEP threshold defaults to 1% instead of 3% (#1580); the default phase budget shares move from 0.40 to 0.38 for FRAMEWORK_AGENT and from 0.50 to 0.47 for KERNEL_AGENT (#1491).

What's included

ROCm 10 and framework stack

• Feat/rocm10 e2e - #1549
• Bump validated SGLang stack to 0.5.20 (ROCm 10 docker) - #1593
• Bring up vLLM on ROCm 10 and fix scheduler stalls found in the 6-leg run - #1605
• Removing patching for SGLang versions - #1486
• fix(profile): follow InferenceX's move of benchmark_serving.py under infx/ - #1594
• fix(profile): keep the profiler bounds out of a preflight rejection, and let the launcher own the trace dir - #1565
• fix(kernel-patch): stop SGLang editable reinstall for in-tree JIT sources - #1567
• fix(aiter): self-heal the compiled registry so a shipped CSV cannot fail every boot - #1532
• fix(provenance): record the AITER that is actually installed - #1508
• refactor(magpie): drop redundant atomic-write patch at pinned MAGPIE_REF - #1592

Enablement phase and replay recipe

• [Stepping] Relocate the enablement subsystem and make ENABLEMENT the sixth phase - #1491
• enablement: make the recipe's replay sufficiency fail closed on what a consumer actually receives - #1414
• fix(warm-replay): replay a KB recipe into a pip-installed framework - #1563

Session lifecycle, limits and recovery

• refactor: retire robustness supervision and simplify benchmark watchdogs - #1547
• fix(mcp): prevent inline action waits from blocking the coordinator - #1577
• refactor(orchestrator): drop five mechanisms with writers but no readers - #1514
• docs(inference_optimizer): detach the optimizer the way the harness understands - #1400

Optimization loop

• [FEAT] dissolve Explore Phase, unify the attempt ledger, judge dryness per lever - #1587
• fix(grid): keep a measurement that served its whole protocol - #1584
• fix(bench): stop baseline benchmarking from a private InferenceX copy - #1456

ATOM

• feat(atom): support Atom end to end — reap its recovery workers, default its kernel backend to forge, and publish it as supported - #1416
• Added support for ATOM detailed profiling - #1509

KernelForge and kernel optimization

• feat(forge): optimize compiler-emitted AMDGPU assembly - #1426
• KernelForge: add fuzzy KB warm-start fallback across framework versions and GPUs - #1512
• Let the rewrite pipeline's caller own its task shape, its files and its stages - #1596
• fix(forge): merge a lane's patch against the KEEPs that landed before it - #1528
• Fix KernelForge patch hygiene and integration ordering - #1583
• fix(kernel): align Forge KEEP state and breakdown attribution - #1522
• fix(forge): do not let a killed campaign's leftovers fail the next task - #1564
• fix(fusion): salvage the campaign keepers a killed run already exported - #1533
• fix(fusion): salvage the published best when a campaign never returns - #1536
• fix(fusion): retry the harness author once when the transport stalls - #1534
• fix: repair kernel tuning promotion and analysis handoffs - #1580
• fix(kernel): preserve operator GEAK checkouts - #1591
• Remove Hyperloom's MI355X achievable roofline fallback - #1602
• fix(roofline): classify capture failures instead of retrying degraded - #1526

Agentic replay (AgentX)

• fix(agentx): unblock baseline admission using profiling error rates - #1568
• fix(agentx): let an agentic recipe satisfy the server-phase pid contract - #1570

Session Breakdown

• feat(sbd-v6): declare the axis a session graded on, and grade the conc sweep on it too - #1527
• feat(sbd): complete the timeline events and pin their wire - #1585
• fix(sbd): give every timeline failure one shape and one place to fail - #1578

Knowledge Base / Recipe

• fix(kb): address one kernel's records by one address, however its source and its environment spell it - #1529
• fix(kb): make a partitioned card a different machine in the KB key - #1542

Agent backend and models

• fix(agents): route every agent CLI choice through the shared rule, on official key names only - #1541
• refactor(llm): one definition of the agent default model ids - #1555

Slurm

• fix(slurm): pass the workload shape to the optimizer CLI as flags - #1571
• fix(slurm): reclaim GPUs from orphaned containers and run docker with --init - #1572
• fix(slurm): stop the GPU reclaim from killing a live neighbour job - #1581
• fix(slurm): do not carry the host's GPU mask into the container - #1582

Docs, skills and review tooling

• chore: retire CHANGELOG.md, and make every README link absolute - #1589
• feat(skills): add the catalog entry skill for Hyperloom bootstrap - #1515
• Add the shared review-pr skill - #1590
• review-pr: make a documented contract violation blocking - #1598
• [DOCS] carry size, complexity and test-boundary rules in review - #1556

CI and tests

• test: keep the both-arms-dry walkthrough off real kernel subprocesses on either route - #1622

Release

• chore(release): cut 1.1.2 - #1619

Three changes reverted before the 1.1.1 cut land in this release: #1526 (re-land of #1511), #1527 (re-land of #1520, with the fail-closed grading path added) and #1542 (re-land of #1495).

Requirements

ROCm 7.2.x or 10.0 platform with a supported AMD Instinct GPU (MI355X / MI325X / MI300X; MI308X detection); Docker with AMD ROCm support, Kubernetes with AMD GPU nodes, or the supported bare-metal setup; Claude Code IDE with Remote SSH; an Anthropic API key or compatible Anthropic gateway, or an OpenAI-compatible gateway; tmux, jq, Node.js >= 18, and Claude CLI.

The recommended stack uses these images: lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920 and lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260920 for SGLang, rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0 for vLLM, and rocm/atom-dev:v0.1.7-rc0 for ATOM (MI355X verified). The images default to /bin/bash, which exits at once in a detached container, so a long-running Hyperloom container has to override the entrypoint (for example --entrypoint tail).

Validated component versions are listed in the compatibility matrix: TraceLens 1.0.0, GEAK 4.0.0, IntelliKit 0.1.1, AgentKernelArena 0.2.0, Magpie 0.2.0.

The wheel is published on PyPI:

pip install hyperloom-inference-optimizer==1.1.2 --target .

Bare-metal installs read HYPERLOOM_WHEEL_TAG, which now defaults to v1.1.2, so this release must carry the rocm-profiler-hotfix-libs.tar.gz asset.

Stability

v1.1.2 is a patch release under semantic versioning. A 1.1.1 session resumes on this build; ensure_schema migrates its coordinator.db. The breaking changes listed above are confined to the CLI surface, the environment contract, the installation prerequisites and the SBD V6 and kernel summary report fields; the wheel's public Python API is unchanged.