Hyperloom v0.7

Highlights

Forge: a third autonomous kernel-optimization backend (new track)

v0.7's headline is Forge (Kernel-Forge) — a self-driving kernel-optimization backend that joins GEAK and OOB. It runs an autonomous edit→build→bench loop with kernel_kind-aware fellow routing (Triton / HIP / CK / aiter / hipBLASLt / FlyDSL), an aiter compiled-kernel closed loop, honest compile-only skips for non-rewritable kernels, and its own session-breakdown lane. Forge already produces the majority of detected kernels on MI300X runs.

Deterministic GEMM tuning (new track)

A standalone forge-gemm-tune backend lands as the default GEMM-tuning prelude: it auto-detects MoE/dense + precision/quant, selects the applicable tuners, and each tuner's tuned config is independently E2E-validated and stacked (per-tuner KEEP/REVERT, like kernel-opt) instead of bundled — so one bad tuner can't drag down the set.

Knowledge Base: GBrain-backed, 7-tuple canonical identity

The Recipe KB extends its canonical identity from a 5-tuple to a 7-tuple with config-donor warm-replay, and Forge fellows now read cross-KB knowledge directly from the unified GBrain (KernelForge + GEAK + PTAO), with KB-usage provenance surfaced in the session breakdown.

Long-horizon durability + specialist autonomy

Long-run optimization is hardened end-to-end: crash-window recovery, integrate-fault retries instead of first-crash discard, duplicate-optimizer corruption protection, orchestrator/CLI decoupling, and opened-up GPU specialist exploration with serving-disjoint leases — so multi-day runs stay productive and self-correcting.

GEAK / kernel-dispatch reliability, end-to-end

A full RCA sweep on the trace→shape→dispatch pipeline: correct GEAK dispatch attribution (no more mis-bucketing non-GEAK attempts), faithful harnesses with real per-arg dtypes, trace-anchored shape pinning, candidate-artifact retry on shared-storage visibility lag, empty-queue clean skip, and forwarding of GEAK scoring/profiler knobs across the Ray boundary.

Profiling / roofline + TraceLens v0.7

A deterministic (no-LLM) trace-analysis route, TraceLens v0.7.0, profile-scoped OSL control with a serialization-safe capture cap, expert-parallel flag handling for MoE roofline, and eager-boot fallback for the SGLang profile-cuda-graph path.

Reliability: sandbox-hang elimination & real-cluster hardening

LLM streaming reads are now bounded client-side and timed-out subprocess trees are reaped by process group, eliminating the "pod Running but idle" sandbox hang. Plus setup_env race / USER_DATA_PATH corruption fixes, user-uncommitted-change protection before destructive reverts, and invalid/premature zero-gain rejection.

CI: structural model pre-filter & throughput

A shared model-compatibility pre-flight (multimodal / Gemma2 / Phi3-longrope / dual-chunk / ModelOpt-FP8 / FlashInfer / gated / missing-tokenizer) skips doomed models before a session is created, alongside HF-token rotation with 429 backoff, larger daily pools, and 48h long-horizon budgets.

What's included in this release

Forge (Kernel-Forge) backend

• Forge autonomous-loop backend for kernel optimization ( #562 )
• Complete Forge backend with RCA fixes + aiter compiled-kernel closed loop ( #613 )
• GBrain KB integration + compiled fellows default-on ( #630 )
• kernel_kind routing + honest compile-only skip + aiter op_test harness fixes ( #644 )
• Forge as its own session-breakdown lane (geak/oob/forge) ( #580 )
• Stability env defaults baked in + auto-recover stale forge branch ( #566 )
• TraceLens-GEAK .py -> .cu resolver ( #626 )

Deterministic GEMM tuning

• Integrate forge GEMM tuning backend ( #637 )
• Make forge GEMM tuning run on post-#637 sessions ( #659 )
• GEMM tuning source attribution + breakdown/artifact cleanup ( #656 )
• Top-level gemm_tuning breakdown section + session_meta exporter fix ( #625 )
• Fix hardcoded a8w8 block-scale GEMM tuning for non-block-scale targets ( #588 )

Knowledge Base

• Extend canonical identity to 7-tuple + config-donor warm-replay ( #587 )
• Optional substrate-KB assessment enrichment for proposals ( #541 )
• KB-usage provenance + decision-process attribution ( #612 )

Long-horizon orchestration & autonomy

• Long-run durability + autonomous-orchestration overhaul ( #546 )
• Harden long-run durability and open up specialist exploration ( #617 )
• Open up GPU specialist autonomy + fix serving-disjoint leases ( #648 )
• STEPPING: bug fixes, optimizations, and orchestrator/CLI decoupling ( #585 )
• Explore optimization — warm rebench, measurement, phase flags ( #552 )
• Retry kernel-integrate faults instead of discarding on first crash ( #582 )
• Prevent a duplicate optimizer from corrupting a session ( #606 / #592 )
• Optimizer stability fixes; prevent invalid & premature zero-gain results; harden abnormal-run handling ( #604 / #649 / #675 )

GEAK / kernel-dispatch reliability

• Correct GEAK dispatch attribution ( #616 / #601 / #602 )
• Faithful GEAK harnesses — real per-arg dtypes + tolerate remapped analysis.md ( #610 / #609 / #611 )
• Forward GEAK scoring/profiler knobs across Ray + aiter harness shape-pinning ( #633 )
• Candidates retry + empty-queue skip ( #653 / #600 / #598 )
• GEAK harness generation, 1-GPU pinning, USER-NTID gateway header ( #591 / #559 / #557 )
• Pin GEAK to v3.2.1 then v3.2.2 ( #579 / #668 )

Profiling / roofline / TraceLens

• Deterministic (no-LLM) trace-analysis route ( #558 )
• TraceLens v0.7.0 ( #663 ); installer for v0.6.1 ( #577 )
• Profile-scoped OSL control + serialization-safe capture cap ( #673 / #571 / #570 )
• vLLM expert-parallel flag for EP-required MoE roofline ( #671 / #569 )
• SGLang roofline capture_traces/*.json.gz handling ( #575 )
• Eager boot on SGLang profile-cuda-graph seq_lens assert ( #632 )
• TraceLens fixes: round-none, version-string patcher, model normalization ( #618 / #572 / #574 )

Observability & session breakdown

• Proposer attribution + operation_kind across timeline & trace ( #561 )
• Record bypass-discovery runs in kernel_journey ( #563 )
• Emit session breakdown to Langfuse on pre-flight fail-fast ( #641 )

Server-arg / framework injection

• Single reliable server-param injection point ( #573 )
• vLLM patch fallback + server-args propagation ( #652 )
• Dedup vLLM attention-backend + preserve GEAK tunnel URL + custom orchestration model ( #547 / #520 / #521 / #340 )

Reliability & real-cluster hardening

• Bound LLM stream reads and reap process groups to stop sandbox idle hangs ( #646 )
• Resolve setup_env.sh race + USER_DATA_PATH corruption ( #629 )
• Stash user changes before destructive git revert ( #627 / #593 )
• Preflight model gates + Gemma2 roofline shape-discovery workaround ( #581 )
• Preflight skips + baseline/roofline stability ( #564 )

CI / Automation

• Structural model-compatibility pre-filter (multimodal/Gemma2/Phi3/ModelOpt/FlashInfer/gated) ( #642 and follow-ups )
• HF-token rotation with 429 backoff ( #614 )
• Daily pool to 10 models; multimodal submit; sglang routing + MI300X TP rebalance ( #560 / #586 / #608 )
• 48h inferencex-daily budget + long-run prompt hints ( #619 ); restore schedule + 6h task budget ( #676 )
• Filter small / not-found models; raise default target gain to 500% ( #603 / #615 / #631 )
• Tiny-workload paths (sandboxy & hyperloom); 32B/80B TP1 ( #636 / #638 / #662 )
• Candidate pools on /wekafs, cron 2x/day ( #568 ); idna bump; CodeQL scan; added unit tests ( #565 / #596 / #584 )

Docs

• Align public docs and KB/backend wording with current runtime ( #583 )
• Upgrade existing docstrings to Google style ( #595 )
• How-to examples and repo usage docs ( #222 )

Requirements

• ROCm platform (tested on MI355X, MI325X, MI300X; MI308X detection)
• Docker with AMD ROCm support, or Kubernetes with AMD GPU nodes
• Cursor IDE with Remote SSH extension (for local workflows)
• LLM API key for GEAK kernel optimization
• OOB API key + base URL for OOB Agent MCP (Claude Code / Codex / Cursor backends)
• GBrain base URL + token for cross-KB knowledge injection (Forge)
• tmux, jq, Node.js >= 18, Claude CLI (@anthropic-ai/claude-code) for the multi-agent runtime