Hyperloom v1.1.0

Hyperloom v1.1.0

This is the first feature release after the 1.0.0 stable release. Kernel optimization now ships in the box: KernelForge is vendored into the Hyperloom wheel as the built-in kernel-opt agent, so there is no separate checkout to clone or install. The optimization loop itself is shorter — configuration search and source landing become two arms of one phase, and the redundant confirmation benchmark after every KEEP is gone.

This release also adds first-class support for agentic-replay workloads: an AgentX run is graded on total token throughput under an interactivity veto, which is the axis a submission is ranked on, and the sweep produces the chart that goes with it. Compute-partition shape (SPX/DPX/QPX/CPX) is now recorded and checked on every session, with a separate operator script to measure which shape a workload wants.

1.1.0 carries a number of breaking changes to the CLI, the environment variables, and the session record. The ones that need a plan before upgrading are listed under Breaking changes; the full per-change list is in CHANGELOG.md.

Highlights

1. KernelForge ships inside Hyperloom

KernelForge's source is vendored into src/kernelforge/ and Hyperloom is the sole source from here on (#1338). Installing Hyperloom installs forge — no private repository to clone, no separate distribution to pip install. Its knowledge base, examples and serving patches ship in the wheel, so they resolve from an installed distribution rather than from a checkout. Kernel selection moves from Hyperloom into KernelForge (#1408) and a nomination contract is added between them (#1366). KERNEL_OPT_BACKEND_ORDER still selects between the forge and geak backends, and eight kernel backends remain (CK, FlyDSL, Triton, Gluon, AITER, HIP, hipBLASLt, and fusion); the intellikit backend is removed because nothing in Hyperloom could reach it.

Vendoring also closed the CLI's tolerance: forge-loop and forge-rewrite-by-flydsl no longer absorb undeclared options (#1343), and every forge CLI option is now documented and pinned to the parser by a test (#1345).

2. One optimization phase, two arms

EXPLORE is merged into FRAMEWORK_AGENT (#1301, #1339), so the chain is PRELUDE → FRAMEWORK_AGENT → KERNEL_AGENT → SWEEP → CLOSE. Configuration search and source/upstream landing are worked in parallel; the phase advances only when both are dry, and one arm plateauing raises switch_bottleneck for the next macro-cycle instead of ending the phase while the other lever still pays. Gain is attributed by lever_kind rather than by phase.

The post-KEEP confirmation round is removed (#1334), the phase/workflow gate layer is dropped (#1393), phase budgets are no longer starved on macro-cycle re-entry (#1358), a roofline target can end a session in SWEEP (#1402, #1406), target_reached routes through CLOSE so the close sequence runs (#1237), and patch delivery carries explicit apply roots end to end (#1300).

3. Agentic replay (AgentX) graded on the axis it is ranked on

The corpus an agentic replay runs averages ~114k prompt tokens against ~810 output tokens per request, so grading on output throughput alone optimizes about 1% of the token budget — a measured baseline read 25978 tok/s total against 183 tok/s output. Total token throughput is now the objective and interactivity p90 (OSL/E2EL) is a veto rather than a weighted term (#1362). SWEEP is reduced to one concurrency sweep, plotted on the axes each workload is ranked by (#1385, #1392), profiling is aligned with the measured workload phase (#1386), round-timeout gaps and overhead-cap visibility are closed (#1309), and the pinned aiperf client is installed and repaired instead of being routed into an enablement round (#1387).

4. Compute-partition awareness

An MI300-series card split into SPX/DPX/QPX/CPX trades per-request latency for aggregate throughput, and until now nothing recorded which shape a number came from. The observed mode goes into the platform fingerprint alongside NPS and the session report names it on partitioned runs (#1288). --compute-partition-mode asserts the mode the card is already in and refuses the session if it is in another one; --streams-per-partition (default 2) sizes the concurrent streams per partition. The optimizer never sets the mode — python3 scripts/partition_mode_sweep.py is where the privileged set lives, measuring each mode in turn and restoring the entry mode on the way out (#1299).

5. Kernel optimization reach

Every hot kernel the trace found now reaches a backend attempt (#1263). Source resolution, the FlyDSL rewrite route and the KERNEL idle guard are unblocked (#1335), and the GEMM lane's three unwired shape inputs — and the five defects behind them — are fixed (#1348). On the GEAK side, handoff provenance is corrected (#1341), the run's actual GPU pin is forwarded in the handoff (#1321), and a run that found nothing is no longer reported as a fault (#1375). An aborted forge-fusion no longer reports as an optimization result (#1359), a kernel_opt that is never dispatched now names the reason (#1363), the fusion manifest read catches what it can actually raise (#1383), and the name heuristic no longer unresolves lane-identified collectives (#1405).

6. Measurement integrity

A framework's JSON args and its measured accuracy are no longer discarded (#1332). The profiler-start transient is trimmed before steady-state splitting (#1292), baseline pre-start cleanup no longer relies on port-health probing (#1361), orphaned GPU memory is reclaimed between roofline retries (#1351), and an unreadable sweep report is marked failed instead of silently passing (#1293). The deterministic trace-analysis route — a second candidate-extraction pipeline beside the one analysis.md defines — is removed in favour of bypass (#1397).

7. Observability and attribution

Session Breakdown V6 records the timeline as it happens and certifies traces at capture time (#1395), covering startup and the Framework Agent (#1331), baseline/sweep/conc_sweep/kernel and close (#1350), and KB events with apply-root paths in agent timelines (#1355). Validated GEAK gains are accounted for in the breakdown (#1340), the gaps that left gateway spend unattributed are closed (#1346), and every outbound LLM call is attributed to the gateway that meters it (#1295).

8. Knowledge Base / Recipe

A published Recipe carries three columns instead of five: config/explore/framework/kernel/patch_timeline collapse to config/patch/kernel, each owned end to end by one SDK facade, and warm replay is keyed to the recorded apply root (#1349). A mid-put crash no longer loses the leftover recipe history envelope (#1310), and PR Monitor reads through the KB Store URL (#1364).

9. Security and governance

Tool-command intel is redacted and non-http gbrain URLs are rejected (#1302); the localization allowlist fails closed and vLLM kv-transfer JSON is serialized correctly (#1306); a retired API key no longer satisfies a credential check (#1382); secret redaction is fixed and accidental agent writes are contained (#1407). Codebase governance guardrails are added (#1370) and the HYPERLOOM_CODEX_EXTERNAL_SANDBOX double opt-in is removed (#1378).

10. Framework and deployment coverage

The recommended SGLang ROCm images move to v0.5.18 (#1304) with a matching TraceLens SHA (#1290), and v2 runner patches are added for vLLM 26/27/28 (#1318). A declared framework venv now outranks the orchestrator's interpreter (#1267), an error-driven restart asks before stopping a prior Hyperloom workload (#1336), and Claude Code __unparsedToolInput emit_intent envelopes are parsed (#1384).

11. Release engineering and CI

A full-path pre-release E2E gate with layered per-leg judging is added (#1303), plus a path-gated KernelForge E2E smoke (#1347), fork-token 403 handling for skipped Forge status (#1357), head_sha-keyed dispatch concurrency (#1367), Forge E2E scoped to loop changes (#1376), and a fix for the e2e dispatcher shadowing the platform orchestration model (#1431).

Breaking changes

An operator upgrading from 1.0.0 has to plan for these:

• EXPLORE is merged into FRAMEWORK_AGENT. --no-explore is removed rather than aliased (use --no-framework-agent). --max-minutes-explore-pct and --phase-budget-explore-pct are aliases for the framework budget option (merged-phase default share 0.40, against 0.50 for KERNEL_AGENT). The explore_* and framework_agent_* exit reasons are replaced by optimize_no_more_leverage, optimize_phase_budget_exhausted and optimize_budget_cap. A session recorded at EXPLORE cannot be resumed by this build — the Coordinator refuses at startup. Archived sessions still read; they just cannot be continued. The framework_agent action is retired (upstream PRs land through integrate_patch with patch_source='upstream_pr', workspaces move from runs/framework_agent/<task_id>/ to runs/integrate_patch/<task_id>/), and pr_intel_specialist is replaced by candidate_discovery_specialist.
• The post-KEEP confirmation round is removed, taking KEEP_UNSTABLE and the stack_rebench_* fields out of the session record.
• The forge CLI rejects undeclared options and exits 2 before any GPU work starts. $FORGE_PATH is removed and not honoured as an override — use $KERNELFORGE_PROJECT_ROOT (defaults to $USER_DATA_PATH/kernelforge, else ~/.cache/hyperloom/kernelforge). forge-gemm-tune is gone as a console script and as a distribution; invoke kernelforge gemm-tune. The kernel-backend vocabulary is normalized (--kernel-backend triton, config key kernel_backend); a config carrying the retired key fails at load rather than migrating silently. FORGE_MAX_ITERS / FORGE_COMPILED_MAX_ITERS are gone.
• The deterministic trace-analysis route is gone. HYPERLOOM_TRACE_ANALYSIS_ROUTE and the analysis_route payload key take agent or bypass; a request still naming deterministic is rejected with invalid_analysis_route. The two tool-free LLM source tiers and HYPERLOOM_LLM_SOURCE_PROVIDER / HYPERLOOM_LLM_SOURCE_PREVIEW are removed in favour of one tool-enabled review session; HYPERLOOM_LLM_SOURCE_MODEL still selects the model.
• SWEEP exit reasons conc_sweep_done / conc_sweep_failed collapse into sweep_done / sweep_failed with no alias.
• Configuration fingerprints are flag-aware, which invalidates every fingerprint persisted in state.json. On the next resume, the session re-benches its explored history.
• A published Recipe carries three columns instead of five; patch_timeline is gone and replay order comes from the zero-padded stack/member indices in each overlay ref.
• Defaults and precedence that changed: the server-boot timeout default is 7200s (up from 2700s); --extra-env now reaches the benchmark for every framework and outranks the config, a precedence change for a custom workload whose YAML sets the same key; launch refuses a session whose streams provably will not fit one partition.

What's included

KernelForge in-tree and the forge CLI

• Vendor KernelForge in-tree as Hyperloom's built-in kernel-opt agent - #1338
• Move kernel selection from Hyperloom into KernelForge - #1408
• Forge nomination contract - #1366
• forge: reject undeclared CLI options instead of absorbing them - #1343
• docs: document every forge CLI option, and pin the doc to the parser - #1345
• Spawn GEMM tuning through the kernel-agents CLI - #1333
• Drop parameters the forge submit path never reads - #1381

One optimization phase, two arms

• Explore + Framework merge: one optimization phase, two arms - #1301
• Finish the explore/framework merge - #1339
• Remove the post-KEEP confirmation round - #1334
• Remove the phase/workflow gate layer and the surface it kept alive - #1393
• Stop starving a phase of budget on every macro-cycle re-entry - #1358
• A roofline target, and a met target that ends in SWEEP - #1402
• Follow-up on #1402: wait for the ladder the met target routed to - #1406
• Route target_reached through CLOSE so the close sequence runs - #1237
• Patch delivery invariant: explicit apply roots end-to-end - #1300
• Remove dead compatibility and fallback paths - #1401

Agentic replay (AgentX)

• Grade an agentic replay on total token throughput - #1362
• One concurrency sweep, plotted on the axes each workload is ranked by - #1385
• Follow-up on #1385: the sweep summary, and what the retired action left behind - #1392
• Align profiling with measured workload phase - #1386
• Close AgentX round timeout gaps + overhead-cap visibility - #1309
• Install the aiperf dependency instead of routing it into enablement - #1387
• Correct the warmup-grace CHANGELOG to what shipped - #1379

Compute-partition awareness

• Record, check and publish the compute-partition shape - #1288
• Partition mode sweep - #1299

Kernel optimization reach

• Get every hot kernel the trace found to a backend attempt - #1263
• Unblock source resolution, the FlyDSL rewrite route, and the KERNEL idle guard - #1335
• GEMM lane: connect the three unwired shape inputs, and fix the five defects behind them - #1348
• Fix GEAK handoff provenance - #1341
• Forward the run's actual GPU pin in the handoff - #1321
• Stop reporting a GEAK run that found nothing as a fault - #1375
• Do not report an aborted forge-fusion as an optimization result - #1359
• Name the reason when kernel_opt is never dispatched - #1363
• Catch what the fusion manifest read can actually raise - #1383
• Stop the name heuristic unresolving lane-identified collectives - #1405

Measurement integrity

• Stop discarding a framework's JSON args and its measured accuracy - #1332
• Trim the profiler-start transient before steady-state splitting - #1292
• Baseline pre-start cleanup no longer relies on port-health probing - #1361
• Reclaim orphaned GPU memory between roofline retries - #1351
• Mark unreadable sweep reports as failed - #1293
• Drop the deterministic trace-analysis route - #1397

Observability and attribution

• Record the SBD V6 timeline as it happens, and certify traces at capture time - #1395
• Add SBD V6 startup and Framework Agent timeline - #1331
• Add SBD V6 baseline/sweep/conc_sweep/kernel timeline and close - #1350
• Add SBD V6 KB timeline events and apply-root paths in agent timelines - #1355
• Account for validated GEAK gains - #1340
• Close the gaps that left gateway spend unattributed - #1346
• Attribute every outbound LLM call to the gateway that meters it - #1295

Knowledge Base / Recipe

• Three-column Recipe SDK with recorded apply roots for warm replay - #1349
• Keep leftover recipe history envelope after a mid-put crash - #1310
• Use KB Store URL for PR Monitor - #1364

Enablement

• Archive what a failed round is actually diagnosed from - #1394
• Add heuristics section to enablement prompt - #1308

Security and governance

• Redact tool-command intel and reject non-http gbrain URLs - #1302
• Fail-closed localization allowlist and serialize vLLM kv-transfer JSON - #1306
• A retired API key must not satisfy a credential check - #1382
• Fix secret redaction and contain accidental agent writes - #1407
• Codebase Governance Guardrails - #1370
• Remove HYPERLOOM_CODEX_EXTERNAL_SANDBOX double opt-in - #1378

Framework and deployment coverage

• Bump SGLang ROCm images to v0.5.18 (20260825) - #1304
• Update the TraceLens SHA for SGLang 0.5.18 patches - #1290
• v2 runner patches for vLLM versions 26, 27, 28 - #1318
• A declared framework venv wins over the orchestrator's interpreter - #1267
• Ask before stopping a prior Hyperloom workload after an error-driven restart - #1336
• Parse Claude Code __unparsedToolInput emit_intent envelopes - #1384

Release engineering and CI

• Add pre-release E2E CI: full-path release gate with layered per-leg judging - #1303
• Add path-gated KernelForge E2E smoke - #1347
• Do not fail Forge skipped-status on fork-token 403 - #1357
• Key E2E workflow_dispatch concurrency on head_sha - #1367
• Scope Forge E2E to loop changes - #1376
• Stop the e2e dispatcher shadowing the platform orchestration model - #1431

Test coverage

• Pin verify_cpp_itfs_rebuilt freshness branches (PRPUNDIT-7) - #1322
• SharedState collective save-failure rollback (PRPUNDIT-8, PRPUNDIT-9) - #1324
• Pin apply_multi_node_invalid_variants CONC filter (PRPUNDIT-10) - #1325
• Pin platform_fingerprint degrade paths (PRPUNDIT-12) - #1327
• Cap _SiteStats.record shape and caller sets (PRPUNDIT-13) - #1328
• Union and dedup prior refs in merge_staged_sections (PRPUNDIT-14) - #1329
• Pin resolve_runtime_dtype and compute-bound ceiling fallbacks (PRPUNDIT-11, PRPUNDIT-15) - #1330
• Pin the _CallStats.first_s sentinel against a 0.0 mutation - #1404

Docs

• README: lead with validated gains and the current harness - #1413
• ROCm docs style guide updates - #1294
• Add a note clarifying the quickstart instructions - #1289
• Fix the 3h demo phase budget and align the skill docs with the CLI - #1417

Release

• Cut 1.1.0 - #1451

Requirements

ROCm platform with a supported AMD Instinct GPU (MI355X / MI325X / MI300X; MI308X detection); Docker with AMD ROCm support, Kubernetes with AMD GPU nodes, or the supported bare-metal setup; Claude Code IDE with Remote SSH; an Anthropic API key or compatible Anthropic gateway, or an OpenAI-compatible gateway; tmux, jq, Node.js >= 18, and Claude CLI.

The recommended stack uses the official upstream images: lmsysorg/sglang-rocm:v0.5.18-rocm724-mi30x-20260825 and lmsysorg/sglang-rocm:v0.5.18-rocm724-mi35x-20260825 for SGLang, and vllm/vllm-openai-rocm:v0.27.1 for vLLM. The vLLM image entrypoint is vllm serve, so a long-running Hyperloom container has to override it (for example --entrypoint tail).

Validated component versions are listed in the compatibility matrix: TraceLens 1.0.0, GEAK 4.0.0, IntelliKit 0.1.1, AgentKernelArena 0.2.0, Magpie 0.2.0.

The wheel is published on PyPI:

pip install hyperloom-inference-optimizer==1.1.0 --target .

Bare-metal installs read HYPERLOOM_WHEEL_TAG, which now defaults to v1.1.0, so this release must carry the rocm-profiler-hotfix-libs.tar.gz asset.

Stability

v1.1.0 is a minor release under semantic versioning. It carries the breaking changes listed above, which are confined to the CLI surface, environment variables, the campaign-config schema and the on-disk session record; the wheel's public Python API and the installation workflow are unchanged.