Hyperloom v0.6

Highlights

Search Space: looser orchestration + long-horizon runs

v0.6 turns the Orchestration theme from v0.5's "search efficiency" to search space: mechanical guardrails are downgraded to advisory so the optimizer drives itself, free-form and cross-domain specialist dispatch lets it explore beyond the fixed action catalogue, and 2–3 day long-horizon optimization with finer-grained start / stop / checkpoint / resume keeps long runs productive and recoverable.

Quantization agent (new track)

A prompt-driven Quark quantization sub-agent lands as an optimization prelude, with Quark enhancements and quantization-agent proposals — quantization joins kernel-opt as a first-class optimization lever.

Knowledge Base: unified and knowledge-graph-backed

cortex-KB and gbrain converge behind a single Recipe KB interface, GBrain is integrated, a Knowledge Graph is wired in, and a KB-evaluation harness is added so warm-start priors can be measured rather than assumed.

GEAK / kernel-optimization reliability, end-to-end

The trace → kernel-shape → GEAK pipeline is hardened so genuinely-good kernels actually reach GEAK and are not silently dropped: cache-invalidation by target type (aiter cpp_itfs / Triton / inductor), recovery of high-GPU-time kernels missed by analysis.md-only extraction, server-patcher idempotency, and parallel GEAK/OOB ladders with per-attempt cache isolation. rocprof-compute roofline and kernel-level roofline detail land alongside.

TraceLens v0.6, CLI + WebUI, and budget-aware roofline

TraceLens v0.6 ships with open-source MAF backfill (GPU microbenchmark), a TraceLens CLI, WebUI standalone/comparative analysis, and roofline that is time-boxed against the total budget.

Multi-node & scale

Multi-node Optimus support comes online and the Arbor mechanism migration is completed.

Observability overhaul

Live Langfuse tracing, full-trace token and conversation logging, phase/step-level observability, a per-session token-consumption breakdown, and an author-time session-breakdown recorder make long runs inspectable in real time.

Reliability & real-cluster hardening

A broad sweep driven by large-scale cluster-run analysis: fast-fail for immediate arg/config errors, log line-buffering so healthy runs no longer look frozen, pod-local dependency roots decoupled from WekaFS, Ray raylet fd-limits, MI308X detection, attention-backend argument hygiene, GEAK container network path to the LLM gateway, and clearer setup/baseline failure classes.

Docs, licensing & coverage

Repo-wide Google-style docstrings with a published Sphinx documentation site, the license relicensed Apache → MIT, requesting-access / SSO docs, and Python test coverage raised to ~91.5%.

What's included in this release

Orchestration & search space

• Orchestration: search space (v0.6 thematic umbrella) — loosen orchestration + free-form / cross-domain specialist dispatch ( #273 )
• 2–3 day long-horizon optimization ( #271 )
• Finer-grained start / stop / checkpoint / resume ( #285 )
• Run-stability follow-ups, stop_reason vocabulary, structural consolidation, and text-coercible multimodal models on the text path

Quantization

• Quark quantization agent as an optimization prelude ( #270 )
• Quark quantization enhancements ( #445 )
• Quantization agent improvement proposals ( #453 )

Knowledge Base

• Unify cortex-KB and gbrain behind one Recipe KB interface ( #455 )
• Integrate GBrain into Hyperloom ( #448 )
• Integrate Knowledge Graph into Hyperloom ( #446 )
• Knowledge Base evaluation ( #447 )

GEAK / kernel optimization

• Invalidate aiter cpp_itfs runtime cache so good kernels aren't discarded ( #458 )
• Generalize integrate-side cache-invalidation by target type (cpp_itfs / Triton / inductor) ( #484 )
• Recover high-GPU-time "other"-bucket kernels dropped by analysis.md-only extraction ( #514 )
• server-patcher: skip already-applied member instead of fail-softing the set ( #505 )
• Inject --moe-runner-backend triton for MoE baselines on AMD ( #512 )
• Honor KERNEL_OPT_BACKENDS in backend selection ( #418 )
• GEAK Ray cross-Python recovery, CUDA-graph attribution, task-group dedupe ( #432 / #431 / #420 )
• Patchability gate hardening + bind GEAK harness to TraceLens-captured shapes ( #381 / #382 )
• Lower the kernel-opt micro speedup KEEP gate to 1.05 ( #442 )
• Kernel-level roofline details ( #406 )

TraceLens / roofline

• Backfill MAF for open-source TraceLens via GPU benchmark ( #390 )
• TraceLens v0.6.0 release integration ( #474 )
• TraceLens CLI support ( #274 )
• TraceLens Agent WebUI standalone / comparative analysis ( #354 )
• Time-box the roofline torch-profiler phase against total budget ( #467 )

Multi-node & scale

• Multi-node Optimus support ( #440 )
• Complete the Arbor mechanism migration ( #450 )

Observability

• Phase/step-level observability in inference_optimizer ( #266 )
• Session token-consumption breakdown ( #443 )
• Live Langfuse push + token_usage breakdown + /workspace artifact packaging; full-trace token + conversation logging + CLOSE bundle

Reliability & real-cluster hardening

• Fast-fail / escalation for subprocess_nonzero arg-error baselines (umbrella) ( #522 )
• Emit reports/final.json on time-exhaust ( #464 )
• Surface the real EngineCore error in the final summary instead of a benign WARN ( #465 )
• Optimizer proposes invalid --attention-backend value → fast reject ( #519 )
• Duplicate --attention-backend flags in launch command ( #520 )
• GEAK container network path to the LLM gateway ( #521 )
• Intermittent SGLang profiler-patch application ( #516 )
• TRACELENS_ROOT not set / TraceLens root not found ( #518 )
• CUDA-graph profiling working-dir ENOENT ( #523 )
• vLLM EngineCore initialization failure surfacing ( #524 )
• Main run log buffers heavily / appears frozen — line-buffer stdout/stderr ( #468 )
• HYPERLOOM_KERNEL_AGENT_ROOT unset stalls optimization ( #469 )
• geak_prompt_patcher upstream-example drift warning ( #470 )
• Requested FP8 precision silently runs BF16 ( #421 / #435 )
• Ray raylet unstable at default fd-limit; needs nofile>=65536 ( #433 )
• MI308X detection + runner mapping ( #376 )
• Clone open-source deps into a pod-local root decoupled from USER_DATA_PATH

Onboarding / Docs / Licensing

• Requesting-access / Primus-SaFE SSO instructions ( #472 )
• Public documentation site ( #222 )
• Repo-wide Google-style docstrings + Sphinx site ( #398 )
• Relicense Apache → MIT
• Raise overall Python test coverage to ~91.5% ( #221 )

CI / Automation

• Daily top-8 model inference-optimization CI ( #452 )
• Sequential optimize cron over the downloads>100 pool
• CI NFS session-liveness + artifact-grace hardening; Pre-flight CI testing ( #473 )

Closed as Not Planned (skipped this cycle)

• baseline_vllm.yaml uses VLLM_ATTENTION_BACKEND env var ( #462 ) — ops-side prestart_patch.py misconfiguration; the env var is a no-op in vLLM and was never selected
• Baseline slow-start / HIP-OOM retry cascade ( #463 ) — root cause was the closed #462; existing reap/recover/retry already covers it, re-scoped to a follow-up "gen-tok/s sanity floor" enhancement
• Check if GEAK already applied patch ( #392 ) — gated on GEAK's patch-application support landing upstream first; revisit once it stabilizes

Requirements

• ROCm platform (tested on MI355X, MI325X, MI300X; MI308X detection added)
• Docker with AMD ROCm support, or Kubernetes with AMD GPU nodes
• Cursor IDE with Remote SSH extension (for local workflows)
• LLM API key for GEAK kernel optimization
• OOB API key + base URL for OOB Agent MCP (Claude Code / Codex / Cursor backends)
• tmux, jq, Node.js >= 18, Claude CLI (@anthropic-ai/claude-code) for the multi-agent runtime