Skip to content

Reframe AgentKernelArena for A/B experiments and improve agent setup#60

Merged
irvineoy merged 5 commits into
mainfrom
docs/ab-testing-rl-ready-refresh
Jul 16, 2026
Merged

Reframe AgentKernelArena for A/B experiments and improve agent setup#60
irvineoy merged 5 commits into
mainfrom
docs/ab-testing-rl-ready-refresh

Conversation

@irvineoy

@irvineoy irvineoy commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Reframe AgentKernelArena as a controlled A/B experimentation platform with RL-ready compilation, correctness, and performance signals.
  • Refresh the README, Sphinx documentation, task/agent guides, security notes, held-out workflow, and visualization copy to match the current repository.
  • Fix onboarding issues with the real clone URL, Node.js/npm prerequisites, native and npm Claude Code installations, config-aware agent checks, and explicit physical-GPU matching guidance.
  • Align task-validator/result-schema guidance and remove stale or inaccurate task and HIP optimization instructions.
  • Add Docker-runner regression coverage for config-selected agent checks, npm-installed Claude Code, explicit AGENTS=all, and internal argument forwarding.

Validation

  • git diff --check
  • bash -n src/scripts/docker_benchmark.sh tests/test_docker_benchmark.sh visualization/setup.sh
  • make check-evaluator (9 tests)
  • make check-perf-helpers
  • Python compile check for modified Python modules
  • YAML parsing for modified configuration and result-template files
  • Local Markdown-link validation across all modified documentation files
  • Make target dry-runs for default and explicit docker-check-agents selection

Comment thread Makefile Outdated

DOCKER_RUNNER := src/scripts/docker_benchmark.sh
CONFIG ?= config.yaml
CONFIG ?= example_configs/benchmark_cursor_mi355x.yaml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new global default run config is a 60-task Cursor benchmark. A first-time user who runs make docker-run with no CONFIG will kick off a large Cursor run. maybe lets try to change default

@irvineoy
irvineoy merged commit 2ba1a1b into main Jul 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants