Home

ACE Wiki

ACE (Agentic Coding Evaluation) is a reproducible framework for evaluating AI coding agents, models, and agent harnesses under controlled conditions — built by StrGm AI Ltd. It drives a coding agent through a frozen, non-trivial build task (a small multi-service trading platform) and scores the result with an independent oracle that never trusts the agent's own word for anything.

ACE's goal is narrower and more practical than a leaderboard: finding which model, harness mode, and hyperparameters actually get a coding task done well — on your own hardware if you're running a local model, or against whichever external provider you've configured. Configure any of those per job in Queue, or run a templated, repeated A/B batch in Experiments, then check Compare and Groups to see which configuration wins for your setup.

This wiki is the reference documentation for ACE: how it's built, why it's built that way, and how to actually use it.

For the project's own top-level docs, see the repo README, docs/DESIGN.md (the authoritative, code-checked design reference), and DEPLOY.md (deploying ace-service + ace-ui-vaadin behind Caddy on a plain VM).

Reference

• Architecture — how ACE is actually built: the two Gradle modules, the job queue and worker model, the benchmark runner (harness modes, agents, context composition, compaction), the scoring oracle and Docker-outside-of-Docker, the LLM provider registry, Preflight, and the Vaadin front end.
• Features — a comprehensive, grounded feature list: the 30-check scoring registry, gaming resistance, Judge, cross-task lessons, operator intervention, the provider registry, Groups/Compare, Experiments, and more.
• Design Decisions — the why behind ACE's non-obvious choices, mined from the codebase's own "why, not what" comments: why SQLite, why one run at a time, why Docker-outside-of-Docker instead of a managed container service, why API keys are stored in plain text, why budget exhaustion is a result and not an invalidation, and more.
• Task Ladder — every task "rung" ACE can run (L1_migration_entity … L7_full_platform, plus the plan-first L3p_point_in_time): what each one asks the agent to build, its default budget, and the exact oracle checks it's scored on.
• Check Registry — every one of the 30 oracle checks (S1…F9), what each actually verifies and how, grounded in its own checker implementation.

User manual

A screenshot-driven walkthrough of the live UI, split by area:

• Jobs — creating a job, the Queue, a running/finished job's live page, and the seven-tab per-task dialog (Prompt, Changes, Restart, History, Judge, Stats, Live).
• Providers & Preflight — configuring an LLM provider (local oMLX or an external OpenAI-compatible provider like Nebius) and running Preflight.
• Results — browsing Runs, reading a Run Detail scorecard, the Groups leaderboard, Compare, and Experiments.

Where to look for what

Question	Page
How does the job queue/worker actually work?	Architecture
What does each oracle check verify?	Check Registry
What does each task rung (L1…L7, L3p) actually ask for?	Task Ladder
Why is Docker-outside-of-Docker used instead of a managed container service?	Design Decisions
How do I create a benchmark job?	User Manual: Jobs
How do I add an LLM provider?	User Manual: Providers & Preflight
How do I compare two runs or two models?	User Manual: Results
How do I deploy ACE?	DEPLOY.md