Repository navigation
AgentKernelArena 0.4.0 makes Arena an RL-ready environment with bounded, independently evaluated rewards. It moves every task to one schema-v2 contract, adds fixed-workload end-to-end model tasks and production-operator suites, and repairs task correctness and benchmark integrity across the suite.
This release tags the existing main commit 47c2658a07efc7c2c0f72751781a5b5690471b0b.
Highlights
Rewards and evaluation
- Replaced the additive point score with a bounded reward in
[0, 1]:max(0, 1 - 1.05 / speedup). Compilation and correctness are prerequisites with no point bonuses, and a correct candidate at or below 1.05× scores zero. See #118 and #137. - Reported agent execution and candidate source changes separately from independent candidate acceptance in results, summaries, CSV export and run comparison. See #114.
- Added an opt-in wall-clock
budget_mode: timefor Claude Code, Cursor, Codex and DeepSeek Harness, and per-taskagent_costs/records with provider-reported cost and token usage. Unreported values arenull, not estimated. See #128.
Unified task contract and new task scopes
- Migrated all retained tasks to one TaskSpec v2
config.yamlcontract. It separates the provided baseline from the candidate, supports initially unimplemented candidates and keeps workloads, numerical comparison and evaluation task-owned. Every integration uses the same candidate and evaluator interfaces. See #107. - Added end-to-end tasks that optimize and fuse kernels across a model's declared sources. Weights, precision, parallelism, serving flags and workloads stay fixed. Evaluation uses paired baseline/candidate serving runs and a read-only candidate verifier. See #116.
- Added 50 Magpie recipes as individually selectable e2e tasks: 45 text, three multimodal and two AgentX. Shared SGLang, vLLM and ATOM adapters back them, with content-addressed runtime packages prepared outside the checkout. See #131.
- Added
sikl_task_builder(make docker-sikl-task-builder), which converts SIKL bundles into self-contained schema-v2 tasks. Supplied workloads, references and comparison rules are preserved. See #96. - Added 46 production-operator tasks under
Aiter-task: BF16 and block-scaled FP8 GEMM, MXFP4 MoE, MLA, mHC and paged top-k, including the DeepSeek-V4 Flash operator set. Also added MI300X FlyDSL BF16 GEMM variants underAiter-task-mi300. See #117, #119 and #136. - Added split-K GEMM, top-k routing, large paged attention, an A8W4 MoE and an SGLang request-to-token pool task. See #75, #58 and #44.
- The tagged source contains 472 task packages: 163
triton2triton, 52e2e, 51triton2flydsl, 49Aiter-task/Aiter-task-mi300, 46torch2flydsl, 41torch2hip, 34hip2hip, 21image_kerneland 15flydsl2flydsl. Availability depends on each task's runtime and architecture contract. A catalog entry is not GPU qualification.
Benchmark integrity and task quality
- Repaired 114 GPU task packages. Changes include narrower editable boundaries, independently drawn timed inputs with replay checks, restored input state and wider correctness coverage. See #127.
- Strengthened correctness and replay checks for further Triton, HIP and FlyDSL tasks. Added rotating timed inputs for AITER GEMM tasks and protected evaluator state in the FlyDSL runtimes. See #113, #115, #120, #121 and #129.
- Added the canonical cold-cache graph protocol, used by the new MI300X GEMM, A8W4 MoE and request-to-token pool tasks. It prepares inputs, runs a read-only 1 GiB eviction sweep, then times one operator graph replay with same-stream GPU Events. Also added an opt-in
cuda_graph_cold_weightsmethod that no task uses yet. See #136 and #125. - The harness guard now rejects added runtime patching and library operator delegation before scoring. See #134.
- Added a
repair_onlyquality-loop mode. Repairs keep the scored contract, need a fresh validator PASS and independent review, and are never reported as speedups. See #129.
Agents and runtime
- Added the opt-in
deepseek_harnessintegration with a pinned headless CLI, API-key authentication, isolated per-task sessions and Docker/Slurm quickstarts. See #111. - Provisioned the pinned Forge engine in Docker (
make docker-setup-forge), with per-revision isolated installations for concurrent campaigns. See #134. - Switched the MI355X (
gfx950) default to the digest-pinned SGLang 0.5.20 / ROCm 10 image. Scoring and resumed workspaces are now bound to their runtime image. See #123. - Added separate MI300X, MI325X and MI300A hardware profiles and quickstarts on shared CDNA 3 (
gfx942) guidance. See #63. - Added
configuration_symbols, which makes the launch settings of symbol-scoped Triton tasks tunable while host wrappers stay protected. It is piloted ontriton2triton/vllm/triton_bmm. See #133. - Reorganized the README and guides around task scopes, a minimal Docker run and reward interpretation (#141). Added repository hygiene rules (#122) and the paper citation.
Upgrade notes
- Scores are not comparable with 0.3.0. The 0.3.0 score was
120 + 100 × speeduppoints for a correct candidate. The 0.4.0 score is bounded in[0, 1], and 2× now scores 0.475. Recompute both arms rather than relabeling historical results. See the scoring reference. - Repaired tasks need fresh baselines. Some tasks changed their timed input distributions or invocation preparation in #121, #127 and #129. Compare scores only within one task revision.
- Task selectors changed. HIP task directories are flat (
hip2hip/gpumode/GELUis nowhip2hip/GELU). Duplicateinstruction2tritontasks were removed, and the unique flip task moved totriton2triton/rocmbench.repositorytasks were removed. Five ROCmBench primitives and 16torch2hip/gpumodegeneration tasks that duplicate retainedhip2hiptasks were retired (#140). Update custom run configs. - Task definitions use schema v2. Migrate custom tasks with the task authoring guide.
- The MI355X runtime changed. To reproduce an older result, select
GFX950_V0514_IMMUTABLE_IMAGEwithdocker_imageorAKA_DOCKER_IMAGEand start a fresh run. Evaluation tools need the matching revision; the ROCm 10 tool profile rejects the old scoring image. - The generic
MI300target no longer assumes MI300X. SelectMI300X,MI325XorMI300Afor the model-specific profile. TheMI325alias remains supported. - Removed the standalone
make vllmlocal model-serving target and its provider documentation. - Reports from earlier revisions lack the agent-execution and candidate-change fields; treat them as unknown.
Validation and limitations
The CI run for the exact release commit passed on the Ubuntu CPU runner with Python 3.12:
make check-perf-helperspython -m pytest -q testspython -m compileall -q src tasks tests
No fresh GPU benchmark or task-validator campaign was run as part of publishing this release. Task additions and repairs carry their own GPU qualification evidence in the linked PRs, scoped to the runtime and source revision recorded there. CPU checks do not establish GPU correctness or performance for every task.
End-to-end tasks need a prepared runtime binding, GPU groups and evaluation budget, as described in the e2e guide. Whole-model tasks currently support the Codex, Claude Code, Cursor and DeepSeek Harness integrations; GEAK and KernelForge adaptation is separate work. Numerical checks and semantic review have finite coverage and do not prove the absence of every invalid implementation.
Evaluation tools remain experimental and are qualified only on MI355X (gfx950). Task workspaces and privileged worker containers are reproducibility boundaries, not security sandboxes. Arena supplies the environment and reward signals; training and policy updates belong to an external system.
Full changelog: v0.3.0...v0.4.0