Hyperloom v0.2
Highlights
TraceLens/OOB/Magpie standalone comes online
Hyperloom now ships dedicated standalone workflows for TraceLens, OOB, and Magpie — each usable independently of the full end-to-end optimization pipeline. The TraceLens Standalone UI supports three input modes — Default (training / non-vLLM/SGLang eager), Inference with Eager (vLLM/SGLang), and Inference with Graph Capture (vLLM/SGLang with a capture folder) — and produces structured analysis reports under /workspace/hyperloom/standalone_analysis.md. The OOB and Magpie standalone paths follow the same invocation pattern, each running its own agent with the appropriate CLI parameters and collecting per-run artifacts accordingly.
Sandbox Queue for user-facing workload scheduling
When a user's Hyperloom request exceeds available cluster resources, the run is no longer rejected — it is placed into a Sandbox Queue, and the user sees their live queue position directly in the UI. This replaces the previous "resource full → fail fast" behavior with deterministic FIFO scheduling against the finite sandbox pool (currently enabled on the oci-slc cluster).
Agent runtime hardening for long-running sessions
The executor's permission model has been rebuilt: the previous wildcard allow-list (which did not match Claude Code built-in tools) is replaced by an explicit allow-list covering Bash, Write, Edit, and other required tools. Agent invocation prompts now enforce strict step order, mandate independent Task subagents for Step 6 and Step 7 (context isolation), and require findings files to follow the shared template. As a result, LLM-heavy categories such as kernel fusion are no longer silently skipped, and subagent hangs caused by permission denial are resolved.
CI/CD with inference A/B testing
Hyperloom ships an inference A/B test workflow integrated into its single-node CI/CD pipeline, allowing optimization proposals to be validated against a baseline automatically before promotion. Combined with the new auto-labeling system (GitHub Models LLM plus a rule-based engine) and project-board automation, the full optimization loop from issue filing to validated CI results is now end-to-end traceable.
Marathon Inference Launcher (3-pane tmux architecture)
Hyperloom introduces a new Marathon Inference Launcher that runs the inference-optimization pipeline as three independent Claude CLI agents in tmux panes (orchestrator, kernel-manager, watchdog), coordinating via JSONL files on shared NFS. This replaces the previous Python harness, provides independent context windows per agent (surviving 24h runs), and supports auto-restart through the CLI --continue loop.
White-box visibility: Root Cause & Pending Cause Agents
Two new supervisory agents make Hyperloom white-box at both ends of the user experience. The Root Cause Agent watches the Marathon optimization loop for failures, diagnoses the cause, and writes actionable guidance back into the next retry prompt — failures stop being binary pass-or-discard and become constrained retries. The Pending Cause Agent does the analogous thing on the queue side: when a user's run is waiting, it surfaces a concrete reason why. Together they replace two previously opaque states — "something failed" and "still waiting" — with explainable, user-controllable answers.
What's included in this release
New integrations and capabilities:
- TraceLens Standalone Analysis UI with three input modes (Default / Inference Eager / Graph Capture)
- TraceLens profiler configuration for quality downstream analysis (steady-state windowing, shape/callstack/graph-capture annotations)
- Hyperloom Local-Local mode for simpler development iteration
- SaFE Plugin, Claw MCP, Magpie Plugin, Sandbox Phase 2, and Sandbox Queue
- Marathon Inference Launcher with the new 3-pane tmux architecture
- Parallel GEAK + LLM kernel optimization race strategy
- GEAK tracing skill for LLM cost attribution
- OOB (Out-of-Box) optimization support
- Claw mode (SaFE RayJob) support in the inference-optimization skill
- MLPerf training optimization skill
- Root Cause / Pending Cause Analysis workstream
CI/CD and automation:
- Single-node inference optimization CI/CD pipeline
- Inference A/B test workflow
- Auto-labeling system combining GitHub Models LLM with a rule-based engine
- Project board automation workflow
- Pinning of IntelliKit to a specific SHA for reproducible Docker builds
Usability and UI:
- Share chat access with the team via link
- Refreshed user-interaction patterns
- Better visibility of the chat 'delete' button on the light theme
- 308x support with configurable model services/clients and user-specified benchmarks
- Clearer code-quality instructions for the patching skill
Bug fixes and stability:
- TraceLens integration now pinned to
release/hyperloom_integration_v0.2; restored generation of outputs such asCustomCollective_fwd.csv - Subagent file-permission denials in the executor resolved via explicit tool allow-list (32 Bash + 16 Write denials eliminated)
- Large file and zip-folder upload supported in TraceLens Standalone UI with automatic decompression on the server
- Removed a source of leaked HF tokens
- BF16 / FP16 precision now supported in addition to FP8 / FP4
- Fixed a case where a requested model was silently replaced by a well-known model
- Fixed false optimization-progress reports when workloads never actually started
- Fixed processes getting stuck during baselining and GEAK tasks
- Fixed multi-node training hang on MI355X after 2 hours
- Resolved session file count inconsistencies and session output UI issues in Standalone Analysis
- Addressed questionable / non-mergeable optimization patches produced by the agent
Requirements
- ROCm platform (tested on MI355X and MI308X)
- Docker with AMD ROCm support, or a Kubernetes cluster with AMD GPU nodes
- Cursor IDE with the Remote SSH extension (for local workflows)
- LLM API key for GEAK kernel optimization
- OOB API key and base URL for OOB Agent MCP (Claude Code / Codex backends)
tmux,jq, Node.js >= 18, and Claude CLI (@anthropic-ai/claude-code) if using the Marathon Launcher