Hyperloom v1.1.4

Hyperloom v1.1.4

This is a patch release on top of 1.1.3. Every command now runs behind one hyperloom <command> entry point. A session lost with its container can be resumed elsewhere, and a failing LLM backend can no longer burn hours of a long run. Hyperloom now measures GPU power and scans server logs for runtime findings, and the validated SGLang stack moves to 0.5.21.

Sessions recorded by 1.1.3 resume on this build. How Hyperloom is launched, installed and configured does change, and so do several report fields; see Breaking changes and the upgrade guide before upgrading. The per-change history lives in docs/release-notes.md.

Highlights

1. One hyperloom command, and no flags that do nothing

hyperloom <command> (or python -m hyperloom <command>) is now the only entry point. setup, check, optimize, recover, quantize, session ... and multi-node ... replace about 50 console scripts and python -m paths, and hyperloom --help lists them all (#1736). Flags that were ignored or only advisory now fail at startup instead of being accepted (#1736, #1739, #1786). On resume, preflight checks the session's recorded framework instead of the SGLang default (#1753), and every --extra-env pin is exported so all readers resolve it alike (#1797, #1818).

2. A session lost with its container can be resumed

hyperloom recover --session-dir S --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT cancels the unfinished tasks of a vanished execution environment and releases their execution and GPU leases, so optimize --resume-from can continue the session elsewhere. Owner scopes now include the start time of the namespace's PID 1, so a new container on the same node is not mistaken for the lost one. A session resumed in SWEEP re-dispatches its lost concurrency sweep, and a crashed integrate_patch whose tree went with the container rolls back while an accepted env-only candidate is replayed into the stack (#1874, carrying #1859, #1870, #1871 and #1879). A local Ray head without the serving_slot resource is restarted with it when that is safe, and unrecoverable baseline failures end the session early (#1816).

3. Long runs stop burning hours on a broken LLM loop

Prompt sections that grew with the session are now bounded; in the reported run this cut a 1.6M-character prompt that failed with Prompt is too long to about 109K (#1735). A failing LLM backend is retried with exponential backoff from 2 s up to 5 min, and an hour without a successful turn stops the session with stop_reason=backend_unhealthy while CLOSE still publishes the best validated stack (#1737). Every Claude SDK turn is bounded by INFERENCE_OPTIMIZER_CLAUDE_TURN_TIMEOUT_SEC (default 1500 s) and its CLI is killed past the bound (#1754). emit_intent is no longer silently hidden from the model, and 50 consecutive turns without an intent also end the session as backend_unhealthy (#1805).

4. Specialists run beside the loop and draw on earlier sessions

The dispatcher no longer waits on running CPU specialists before the tick advances, and the top untested specialist proposals are benched automatically as one explore grid (#1742). Each workspace runs a local Experience KB service: measured FRAMEWORK_AGENT attempts are published as Experiences and read back into prompts, with optional sync against a shared global KB (#1732). Operators can define their own specialist with --custom-specialist-prompt-file and --custom-specialist-description (#1804). The installer owns the single Claude CLI on PATH, and specialists always run as CLI subprocesses (#1775, #1786).

5. Runtime findings from server logs, and verdicts that say why

Every measured server log is scanned for runtime findings such as unknown vLLM environment variables, disabled fusions and AITER tuned-config misses; they are written to runtime_findings.json and shown to every role. A patch that verifiably resolves a correctness finding may KEEP with up to a 3% drop on the graded axis, recorded as keep_reason=correctness_fix rather than as a gain (#1793). AgentX REVERT reasons name every gate that refused a candidate (#1807), guard-refused explore variants show their graded movement instead of no_meas (#1819), and explore skips variants whose changes are already in effect on the running server (#1647).

6. GPU power is measured, and scriptable workloads gain a latency ceiling

Per-GPU power is sampled through amd-smi over the measured phase only and recorded as gpu_power.json and gpu_power_avg_w; HYPERLOOM_GPU_POWER_SAMPLING=0 turns this off. --gpu-power-cap-w and --gpu-perf-level assert, and never set, the GPU power settings at launch and on resume (#1730). For xdit and custom workloads, --max-latency-ms makes mean end-to-end latency a constraint on every KEEP, and a baseline already over the ceiling stops with baseline_over_latency_budget (#1297).

7. AgentX: MLPerf Endpoints client, one launch path, fairer comparisons

HYPERLOOM_AGENTIC_BACKEND=mlperf swaps in the MLCommons agentic harness, graded on output throughput; it measures Kimi-K3 only and accepts any registered serving framework for it (#1668, #1771). The agentic-recipe launch path is removed: AgentX boots the server through the generic InferenceX launcher (#1784). GLM-5.3 replays the full InferenceX corpus instead of the 256K-capped one (#1796). Explore KEEPs persist their graded axes so they are no longer dropped at promotion (#1790, #1806), and a timed-out or cancelled round stops the server its AgentX client booted instead of leaving it on the port and GPU memory (#1874).

8. Profiling and kernel targeting follow TraceLens directly

The kernel agent reads TraceLens' structured analysis.json and resolve_kernel_source instead of scraping analysis.md (#1671), and the Forge kernel_opt queue follows TraceLens' priority order (#1763). Per-batch-size CUDA-graph capture works on SGLang >= 0.5.18 (#1745), spawned vLLM engines keep their GPU events in profile traces (#1777), and the roofline profiles once per configuration and records a profile only together with its analysis (#1830, #1836). Missing GEMM tuning CLIs are reported at preflight and in GEAK results (#1780), and KernelForge adds kernelforge roofline-ceiling to estimate each shape's achievable latency and steer a campaign against it (#1603).

Breaking changes

An operator upgrading from 1.1.3 has to plan for these. The full old-to-new command table is in the upgrade guide.

• Every command runs under hyperloom <command> (#1736). The inference_optimizer and quantization-agent console scripts are removed, and so are python -m hyperloom.inference_optimizer.{cli,multi_node,setup}. Running hyperloom alone used to run setup; it now prints usage and exits 2, so use hyperloom setup. recover-session becomes hyperloom recover, the python -m ...tools.* session tools become hyperloom session {breakdown,report,backfill,events,state}, and --verbose goes after the command name. kernelforge stays separate.
• Removed flags fail with unrecognized arguments:
  • --target-summary, --warm-replay-min-reproduce-pct, --plateau-kernel-{revert-streak,keep-gain,lookback} (#1739);
  • --specialist-dispatch-mode, --specialist-per-turn-max-seconds (#1786);
  • --kernel-codex, --kernel-claude, --kernel-prompt, recover --backfill-trace (use hyperloom session backfill) (#1736);
  • --max-minutes-<phase>-pct and --phase-budget-explore-pct; use --phase-budget-<phase>-pct (#1736).
• A failed --quantize prelude exits 4, not 3 (#1736). Exit 3 now only means another optimizer holds the session lock.
• The installer provides one claude CLI, and specialists require it (#1786). With no claude on PATH, the installer runs Claude Code's native installer, so a host without curl or access to claude.ai now fails the install. The installer no longer installs Node.js; images that use the Codex carrier must provide node and npm. A session whose specialists cannot find claude stops at startup.
• The kernel tools ship inside the Hyperloom package; re-run install.sh after upgrading (#1787, #1829). HYPERLOOM_KERNEL_AGENT_ROOT and KERNEL_AGENT_ROOT are no longer read, install.sh --skip-kernel-agent is removed, and preflight exits 2 when $KERNEL_AGENT_ENV (default $USER_DATA_PATH/runtime/kernel-agent.env.sh) is missing.
• More environment values are parsed strictly (#1739). A non-numeric value now raises EnvValueError instead of silently taking the default, for INFERENCE_OPTIMIZER_CYCLE_HOURS, INFERENCE_OPTIMIZER_CYCLE_RELOOP_MIN_REMAINING_SEC, INFERENCE_OPTIMIZER_BACKEND_ERROR_STREAK_THRESHOLD, INFERENCE_OPTIMIZER_REACTOR_TURN_TIMEOUT_SEC, INFERENCE_OPTIMIZER_KERNEL_IDLE_MAX_TICKS, INFERENCE_OPTIMIZER_KERNEL_IDLE_MIN_SECONDS, SWEEP_VARIANT_TIMEOUT_SEC, HYPERLOOM_FUSION_KEEP_PCT, HYPERLOOM_GEMM_PAIRED_PAIRS and GEAK_{E2E_TIMEOUT_S,BUDGET_MARGIN_S,MIN_RUN_S,TERM_GRACE_S}. INFERENCE_OPTIMIZER_INLINE_FAST_ACTIONS and INFERENCE_OPTIMIZER_INLINE_ACTION_TIMEOUT_S are removed.
• The AgentX agentic-recipe launch path is removed (#1784). An AGENTX_SERVER_SCRIPT pointing at an InferenceX single_node/agentic/*.sh recipe exits 2 at server boot; move its server args into --server-args and its env into --extra-env.
• KernelForge no longer reads older workspace formats (#1768). forge-loop --resume on a workspace written by an older build fails; start a fresh campaign. forge-rewrite-by-flydsl --op-name becomes --logical-op-name, and provider plugins must register under kernelforge.agent_providers.
• Workload knobs in --extra-env and the shell now take effect (#1797, #1818). Every --extra-env pin is exported under its own name, so a launch script that pinned TP=4 or ISL=4096 that way now runs at that shape instead of the default. On a fresh launch, ISL, OSL, CONC, TP, EP and PRECISION exported in the shell, which 1.1.3 ignored, also take effect; unset any you did not mean. On --resume-from the session's recorded shape wins over stale shell exports, except that an exported PRECISION still overrides the recorded precision, so unset it before resuming.
• The validated SGLang images move to 0.5.21 (#1795): lmsysorg/sglang-rocm:v0.5.21-rocm10-{mi30x,mi35x}-20261008.
• An EP-size pin in the server args that contradicts --ep now fails at materialize (#1809). Drop the pin or pass a matching --ep.

Defaults that change without operator action:

• AgentX sessions skip the post-optimization concurrency sweep unless --enable-conc-sweep is passed (#1659).
• A session stops with stop_reason=backend_unhealthy after an hour of LLM backend failures or 50 turns without an intent (#1737, #1805).
• A local legacy warm replay is held to the session's KEEP threshold rather than any positive gain, because warm replay now honours graded adoption (#1739). AgentX candidates are also judged on request error rate, with 0.5 pp of slack on rounds of at least 200 requests (#1668, #1807).
• GLM-5.3 AgentX baselines replay the uncapped corpus and are not comparable with 1.1.3 runs (#1796). To keep the old corpus, set AGENTX_CANONICAL_DATASET=semianalysis_cc_traces_weka_062126_256k; pinning it through AGENTX_DATASET instead marks every result submission_valid=false.
• A resume that passes a different --framework exits 2 (#1753).

Report and JSON consumers also need these updates:

• state.json drops target_summary, warm_replay_min_reproduce_pct, current_action, target_gap_pct, budget_extensions, orchestration_memory_history, cycle_directive_history and several other unread fields, and the UPDATE_STATE intent is removed (#1723, #1739).
• optimization_journal.json gains the outcome recorded for specialist, report, profile and roofline rows, and the SBD V6 phase_action settle decision becomes adopted / refused / reverted / failed / recorded instead of promoted / no_promote (#1739).
• warm_replay_outcome and the SBD warm_replay verdict row drop below_historical_reproduce_pct and historical_reproduce_bar_pct (#1739).
• New sessions publish the hyperloom.workflow_evaluation.v2 contract; sessions started on 1.1.3 keep v1 (#1733). The v1 schema drops prelude_inputs.double_run and adds the stop reasons backend_unhealthy and baseline_over_latency_budget, so its contract_digest changes (#1297, #1737, #1739).
• A cancelled concurrency sweep records status: "cancelled" instead of sweep_failed (#1767), and a session that reaches skip_to_close without a measured baseline ends as baseline_failed (#1751).
• Roofline profiles a configuration once, so max_attempts is 1 and the retry_after_* reasons are gone (#1830).
• reports/enablement/patches/ and reports/enablement/artifacts/ are no longer written; per-round copies under reports/enablement/<task_id>/ remain (#1789).

What's included

CLI, setup, preflight and resume

• fix: honor configured Ray versions during CLI preflight - #1516
• refactor(cli): one hyperloom <command> entry point; remove dead CLI shells and fix CLI bugs - #1736
• Detach Ray daemons from the caller's session and stdio - #1752
• fix(cli): resume preflight checks the session's framework, not the default - #1753
• fix(install): find the image Python in /opt/python/bin - #1774
• fix(specialists): spawn the Claude CLI the installer recorded - #1775
• fix(preflight): expand ${VAR} references in the custom-header settings - #1776
• fix(specialists): one claude CLI on PATH, and specialists run only as CLI subprocesses - #1786
• Bump validated SGLang stack to 0.5.21 (ROCm 10) - #1795
• fix(cli): export every --extra-env pin so all readers resolve it alike - #1797
• Repair a local Ray head without serving_slot; stop early on unrecoverable baseline failures - #1816
• fix(cli): let a resumed session's recorded workload outrank the shell - #1818
• Run the kernel agent as a Hyperloom module instead of a standalone tool - #1829
• release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871, #1879) and round server teardown (partial #1606) - #1874

Orchestration, specialists and long-run reliability

• [Bad smell] fix(loop): compose the Coordinator from its collaborators instead of delegating by name - #1649
• [Bad smell] fix: keep AgentX sessions off the conc sweep, and correct comments that contradict the code - #1659
• fix(integrate): refuse a git tree with no readable HEAD before touching it - #1713
• fix(orchestrator): preserve proposal provenance across SBD and Langfuse - #1714
• fix(baseline): restore a warm-replay tree through the backup ledger - #1715
• fix(context-tools): let context reader failures reach the logging guard - #1718
• fix(orchestrator): unwind a stack checkpoint from its apply rows - #1721
• refactor(explore): drop the server --help probe from the compatibility filter - #1722
• refactor(policy): gate UPDATE_STATE on one whitelist - #1723
• refactor(orchestrator): settle a stack checkpoint from one entry - #1727
• fix(orchestrator): bound the prompt sections that grow with the session - #1735
• fix(coordinator): back off a failing LLM backend and stop after an hour - #1737
• [Refactor] Orchestration layer: retire delegation mechanism and converge ownership - #1739
• [Feat] decouple CPU specialists from the tick--Async Specialist - #1742
• fix(orchestrator): close an unmeasured session as baseline_failed on skip_to_close - #1751
• fix(orchestrator): bound every Claude SDK turn by wall clock and kill its CLI - #1754
• fix(sweep): stop a cancelled conc_sweep, and let SWEEP wait for the sweep it granted - #1767
• fix(prelude): keep PRELUDE open until the initial roofline books - #1779
• feat(specialist): operator-defined custom specialist type - #1804
• fix(orchestrator): stop a hidden emit_intent from silently stalling a session - #1805
• fix(orchestrator): keep a PRELUDE skip_to_kernel hint from ending FRAMEWORK_AGENT - #1808
• fix(orchestrator): size GPU action leases from the session's expected cost - #1837

Measurement, grading and explore

• feat(keep): --max-latency-ms as a constraint on every promotion - #1297
• feat(explore): skip baseline-noop variants - #1647
• fix(kv-metrics): honor AIPerf phase ranges - #1700
• feat(power): measure per-GPU power over the measured phase; assert and record the GPU power settings - #1730
• feat(runtime-findings): scan server logs, surface findings to every role, allow verified correctness-fix KEEP - #1793
• fix(executors): honour --ep on single-node servers through the argv seal - #1809
• fix(executors): keep the inherited launch paths at every prepend site - #1810
• fix(explore): stop rendering guard-refused variants as unmeasured - #1819

AgentX

• feat(agentx): opt-in MLPerf Endpoints agentic client for AgentX - #1668
• [Feat] Deliver config levers to AgentX agentic recipes; stop injecting launcher defaults into them - #1696
• fix(agentx): write the agentic result into RESULT_DIR - #1749
• feat(agentx): let the MLPerf agentic flow run on any serving framework - #1771
• refactor(agentx): remove the agentic-recipe launch path - #1784
• fix(agentx): persist graded axes on explore KEEP for lift - #1790
• fix(agentx): replay the full corpus for GLM-5.3, matching InferenceX - #1796
• fix(agentx): derive explore winner axes, park baseline aiter drops off NFS, sandbox aiter in tests - #1806
• fix(agentx): name the refusing gate, tolerate half a point of extra failed requests - #1807
• fix(agentx): let the server-side step bound end a bounded profile capture - #1831

Enablement

• [Bad smell] fix(executors): stop echoing credentials in rejected setup commands, and give the setup replay its own modules - #1704
• [Bad smell] fix(framework): resolve the xDiT source tree and derive per-framework facts from the registry - #1705
• [Bad smell] fix(enablement): keep an earlier patch-root binding at KEEP, and let EnablementRound own its stack records - #1706
• fix(enablement): fail closed when the KEEP cannot read its framework tree or materialized config - #1728
• Fix: give the dispatched failure signature one owner, reaching rungs 3 and 4 - #1746
• fix(enablement): key boot failures on their root cause, unblock PR discovery; roofline fixes - #1750
• fix(enablement): make enablement_setting.sh reproduce the setup on its own - #1789
• fix(enablement): keep the revalidation window open until its result is booked - #1820
• fix(enablement): judge vLLM launches with JSON/enum args on what they ran - #1821
• fix(enablement): the setup/boot/grade path of an enablement round - #1841

Kernel, profiling, roofline and KernelForge

• Create a script to generate kernel_candidates.json from the TraceLens Analysis.md - #1574
• feat(roofline): establish each shape's achievable latency, then steer and stop on it - #1603
• Re-Architect TraceLens Interface in Hyperloom - #1671
• fix(roofline): size experts by the precision they are stored at - #1726
• add patching for gc - #1745
• Re-Order Tasks by TraceLens Sorting Order - #1763
• Read kernelforge workspace state only in the shape the current code writes - #1768
• fix(gemm-tune): keep the TunableOp recording envs in the vLLM shape capture - #1772
• fix(forge): ignore rocminfo's generic ISA names when detecting the GPU target - #1773
• fix(profile): keep GPU events in the traces of spawned vLLM engines - #1777
• feat(kernel): surface missing kernel-tuning tools at preflight and from GEAK - #1780
• fix(phases): treat a GEAK timeout as a terminal KERNEL result - #1783
• chore(kernelforge): retire obsolete SGLang serving patch - #1785
• fix(roofline): integer AgentX phase-wait cap, profile once, honour LLM probe timeout - #1830
• fix(roofline): record the profile on shared state only together with its analysis - #1836

Knowledge, reporting and workflow evidence

• fix(breakdown): retain rows with opaque natural identifiers - #1559
• fix(trace): stop a resumed leg re-pushing every backfill row to Langfuse - #1657
• fix(kb): read baseline server args under the baseline's own framework - #1712
• Experience KB: in-repo service per workspace, prompt reads, and global sync - #1732
• feat(workflow-eval): publish v2 evidence and CLOSE receipts - #1733
• fix(workflow-eval): record CLOSE before terminal task dispatch - #1741
• fix(orchestrator): freeze current_balance only when the session contract declares it - #1755
• fix(breakdown): mask credentials in the exported launch env - #1756
• feat(tools): kbmine, a standalone no-run uplift estimator over the Recipe KB and Pulse - #1778

Internal architecture and refactors

• [Bad smell] refactor: give misleading names the meaning their code has - #1658
• [Bad smell] refactor: replace the long parameter lists in conc_sweep and bypass_runner with parameter objects - #1660
• chore(env): remove env vars that nothing reads - #1692
• [Bad smell] refactor(roofline): lift the profile retry loop out of _execute - #1702
• [Bad smell] refactor(integrate_patch): move the enablement KEEP evidence helpers out of the executor - #1703
• [Bad smell] refactor(round_store): take the held Round, and give each round write one owner - #1707
• test(delivery): verify restored trees against their recorded digest - #1716
• refactor(delivery): delete the nogit revert the ledger restore replaced - #1717
• refactor(state): return the lease tables to the modules that own them - #1719
• refactor(dispatcher): keep the reconcile key convention in one module - #1720
• Remove duplicate GPU resolution from CLI model gates - #1725
• [Relocate] Kernel agent: move the kernel tools to their owner packages and retire HYPERLOOM_KERNEL_AGENT_ROOT - #1787
• refactor(kernelforge): restructure knowledge into domain packages - #1788
• [Bad smell] refactor(kernel): keep each kernel-tool rule with its owner - #1791
• [Relocate] Coordinator layer: move code to the collaborator that owns it - #1794

CI, review tooling and docs

• ci(e2e): dispatch through Dispatron, keep the reporting here - #1489
• ci(forge-e2e): dispatch through Dispatron, keep the reporting here - #1496
• ci: grade the federated skill's routing in this repo - #1708
• docs: state Codex support where the docs name only Claude - #1740
• ci(e2e): report a run that never got a GPU as no capacity - #1747
• chore(deps): bump actions/checkout from 4 to 7 - #1758
• chore(deps): bump actions/upload-artifact from 4 to 7 - #1759
• docs(profile): scope the start-up shim's note to the stack it was observed on - #1799
• review-pr: add D11 and D12, and make complexity above 20 a gate - #1811
• ci: code-metrics gate on touched files with baseline-free checks; fold in import-linter, PR hygiene and ruff principle rules - #1812
• docs(review-pr): add the A rule family for code shape, and the AGENTS.md principles it rests on - #1815
• review-pr: add T5, X8, D13, V7 and V8 from #1797 - #1817
• test: stop the serving_slot degrade test passing on another test's leftovers - #1842

Release

• chore(release): cut 1.1.4 - #1905

Requirements

AMD Instinct MI300X, MI325X or MI355X; Ubuntu 24.04 recommended or Ubuntu 22.04 legacy; ROCm 7.2.x or 10.0; Python >= 3.10; Docker with AMD ROCm support, Kubernetes with AMD GPU nodes, or the supported bare-metal setup; Claude Code IDE with Remote SSH; an Anthropic API key or compatible Anthropic gateway, or an OpenAI-compatible gateway; tmux, jq, and curl with access to claude.ai. The installer provides the Claude CLI when none is on PATH; Node.js is needed only for the Codex carrier and must come from the image.

The validated images are lmsysorg/sglang-rocm:v0.5.21-rocm10-mi30x-20261008 (MI300X / MI325X) and lmsysorg/sglang-rocm:v0.5.21-rocm10-mi35x-20261008 (MI355X) for SGLang, rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0 for vLLM, and rocm/atom-dev:v0.1.7-rc0 for ATOM on MI355X. The images default to /bin/bash, which exits at once in a detached container, so a long-running Hyperloom container has to override the entrypoint (for example --entrypoint tail).

Validated component versions are listed in the compatibility matrix: TraceLens 1.0.0, GEAK 4.0.0, IntelliKit 0.1.1, AgentKernelArena 0.2.0, Magpie 0.2.0.

The wheel is published on PyPI:

pip install hyperloom-inference-optimizer==1.1.4 --target .

Bare-metal installs read HYPERLOOM_WHEEL_TAG, which now defaults to v1.1.4, so this release must carry the rocm-profiler-hotfix-libs.tar.gz asset.

Stability

v1.1.4 is a patch release under semantic versioning. Sessions recorded by 1.1.3 resume on this build: the coordinator database schema and the SBD wire version are unchanged, state.json loads with removed keys dropped, and a 1.1.3 session keeps its v1 workflow contract. A KernelForge workspace written by 1.1.3 cannot be continued with forge-loop --resume, and downgrading a 1.1.4 session to 1.1.3 is not supported. The breaking changes are confined to the command-line entry points and flags, installer and Claude CLI provisioning, kernel-tool paths, strict environment parsing, the AgentX agentic-recipe path, KernelForge workspace formats, the SGLang images and the report fields listed above.