Skip to content

AgentKernelArena 0.4.0

Latest

Choose a tag to compare

@irvineoy irvineoy released this 03 Nov 00:00
47c2658

AgentKernelArena 0.4.0 makes Arena an RL-ready environment with bounded, independently evaluated rewards. It moves every task to one schema-v2 contract, adds fixed-workload end-to-end model tasks and production-operator suites, and repairs task correctness and benchmark integrity across the suite.

This release tags the existing main commit 47c2658a07efc7c2c0f72751781a5b5690471b0b.

Highlights

Rewards and evaluation

  • Replaced the additive point score with a bounded reward in [0, 1]: max(0, 1 - 1.05 / speedup). Compilation and correctness are prerequisites with no point bonuses, and a correct candidate at or below 1.05× scores zero. See #118 and #137.
  • Reported agent execution and candidate source changes separately from independent candidate acceptance in results, summaries, CSV export and run comparison. See #114.
  • Added an opt-in wall-clock budget_mode: time for Claude Code, Cursor, Codex and DeepSeek Harness, and per-task agent_costs/ records with provider-reported cost and token usage. Unreported values are null, not estimated. See #128.

Unified task contract and new task scopes

  • Migrated all retained tasks to one TaskSpec v2 config.yaml contract. It separates the provided baseline from the candidate, supports initially unimplemented candidates and keeps workloads, numerical comparison and evaluation task-owned. Every integration uses the same candidate and evaluator interfaces. See #107.
  • Added end-to-end tasks that optimize and fuse kernels across a model's declared sources. Weights, precision, parallelism, serving flags and workloads stay fixed. Evaluation uses paired baseline/candidate serving runs and a read-only candidate verifier. See #116.
  • Added 50 Magpie recipes as individually selectable e2e tasks: 45 text, three multimodal and two AgentX. Shared SGLang, vLLM and ATOM adapters back them, with content-addressed runtime packages prepared outside the checkout. See #131.
  • Added sikl_task_builder (make docker-sikl-task-builder), which converts SIKL bundles into self-contained schema-v2 tasks. Supplied workloads, references and comparison rules are preserved. See #96.
  • Added 46 production-operator tasks under Aiter-task: BF16 and block-scaled FP8 GEMM, MXFP4 MoE, MLA, mHC and paged top-k, including the DeepSeek-V4 Flash operator set. Also added MI300X FlyDSL BF16 GEMM variants under Aiter-task-mi300. See #117, #119 and #136.
  • Added split-K GEMM, top-k routing, large paged attention, an A8W4 MoE and an SGLang request-to-token pool task. See #75, #58 and #44.
  • The tagged source contains 472 task packages: 163 triton2triton, 52 e2e, 51 triton2flydsl, 49 Aiter-task/Aiter-task-mi300, 46 torch2flydsl, 41 torch2hip, 34 hip2hip, 21 image_kernel and 15 flydsl2flydsl. Availability depends on each task's runtime and architecture contract. A catalog entry is not GPU qualification.

Benchmark integrity and task quality

  • Repaired 114 GPU task packages. Changes include narrower editable boundaries, independently drawn timed inputs with replay checks, restored input state and wider correctness coverage. See #127.
  • Strengthened correctness and replay checks for further Triton, HIP and FlyDSL tasks. Added rotating timed inputs for AITER GEMM tasks and protected evaluator state in the FlyDSL runtimes. See #113, #115, #120, #121 and #129.
  • Added the canonical cold-cache graph protocol, used by the new MI300X GEMM, A8W4 MoE and request-to-token pool tasks. It prepares inputs, runs a read-only 1 GiB eviction sweep, then times one operator graph replay with same-stream GPU Events. Also added an opt-in cuda_graph_cold_weights method that no task uses yet. See #136 and #125.
  • The harness guard now rejects added runtime patching and library operator delegation before scoring. See #134.
  • Added a repair_only quality-loop mode. Repairs keep the scored contract, need a fresh validator PASS and independent review, and are never reported as speedups. See #129.

Agents and runtime

  • Added the opt-in deepseek_harness integration with a pinned headless CLI, API-key authentication, isolated per-task sessions and Docker/Slurm quickstarts. See #111.
  • Provisioned the pinned Forge engine in Docker (make docker-setup-forge), with per-revision isolated installations for concurrent campaigns. See #134.
  • Switched the MI355X (gfx950) default to the digest-pinned SGLang 0.5.20 / ROCm 10 image. Scoring and resumed workspaces are now bound to their runtime image. See #123.
  • Added separate MI300X, MI325X and MI300A hardware profiles and quickstarts on shared CDNA 3 (gfx942) guidance. See #63.
  • Added configuration_symbols, which makes the launch settings of symbol-scoped Triton tasks tunable while host wrappers stay protected. It is piloted on triton2triton/vllm/triton_bmm. See #133.
  • Reorganized the README and guides around task scopes, a minimal Docker run and reward interpretation (#141). Added repository hygiene rules (#122) and the paper citation.

Upgrade notes

  • Scores are not comparable with 0.3.0. The 0.3.0 score was 120 + 100 × speedup points for a correct candidate. The 0.4.0 score is bounded in [0, 1], and 2× now scores 0.475. Recompute both arms rather than relabeling historical results. See the scoring reference.
  • Repaired tasks need fresh baselines. Some tasks changed their timed input distributions or invocation preparation in #121, #127 and #129. Compare scores only within one task revision.
  • Task selectors changed. HIP task directories are flat (hip2hip/gpumode/GELU is now hip2hip/GELU). Duplicate instruction2triton tasks were removed, and the unique flip task moved to triton2triton/rocmbench. repository tasks were removed. Five ROCmBench primitives and 16 torch2hip/gpumode generation tasks that duplicate retained hip2hip tasks were retired (#140). Update custom run configs.
  • Task definitions use schema v2. Migrate custom tasks with the task authoring guide.
  • The MI355X runtime changed. To reproduce an older result, select GFX950_V0514_IMMUTABLE_IMAGE with docker_image or AKA_DOCKER_IMAGE and start a fresh run. Evaluation tools need the matching revision; the ROCm 10 tool profile rejects the old scoring image.
  • The generic MI300 target no longer assumes MI300X. Select MI300X, MI325X or MI300A for the model-specific profile. The MI325 alias remains supported.
  • Removed the standalone make vllm local model-serving target and its provider documentation.
  • Reports from earlier revisions lack the agent-execution and candidate-change fields; treat them as unknown.

Validation and limitations

The CI run for the exact release commit passed on the Ubuntu CPU runner with Python 3.12:

  • make check-perf-helpers
  • python -m pytest -q tests
  • python -m compileall -q src tasks tests

No fresh GPU benchmark or task-validator campaign was run as part of publishing this release. Task additions and repairs carry their own GPU qualification evidence in the linked PRs, scoped to the runtime and source revision recorded there. CPU checks do not establish GPU correctness or performance for every task.

End-to-end tasks need a prepared runtime binding, GPU groups and evaluation budget, as described in the e2e guide. Whole-model tasks currently support the Codex, Claude Code, Cursor and DeepSeek Harness integrations; GEAK and KernelForge adaptation is separate work. Numerical checks and semantic review have finite coverage and do not prove the absence of every invalid implementation.

Evaluation tools remain experimental and are qualified only on MI355X (gfx950). Task workspaces and privileged worker containers are reproducibility boundaries, not security sandboxes. Arena supplies the environment and reward signals; training and policy updates belong to an external system.

Full changelog: v0.3.0...v0.4.0