Repository navigation
Check Registry
oracle/CheckId.java is the single enum source of truth for every check the oracle can score: 30
ids, each with a fixed category, weight, and one-line description. A task rung scores
only a fixed subset of these 30 — see each rung's own Checks: line there, every id linked back
to its own entry here. This page is the detailed reference: what each id actually verifies and how,
grounded directly in the checker that implements it, not just the one-line description.
How a result resolves into a score — CheckStatus's five values, weights, the fixed denominator, and
why a SKIPPED/INFRA result still blocks a clean 100% — is covered in
Features: Scoring; this page assumes that
and focuses on each check's own mechanics. One invariant worth repeating here because it shapes every
section below: RunOracle.score() builds its result set as wanted check → record, and any wanted
id with no record becomes a FAIL — a checker that crashes, times out, or simply never emits
something can never raise a score by going silent.
All nine run offline (no Docker), as pure text/regex/filesystem checks over the workspace — weight 1
each. StructureChecks.java is the implementation.
docs/TASK_DEFINITION.md exists and is over 200 bytes — a touched empty file doesn't pass.
docs/IMPLEMENTATION_PLAN.md exists and is over 200 bytes, same rule as S1.
docs/PROGRESS.md exists and is over 200 bytes, same rule again.
A docker-compose file is present. When more than one exists, the root-most one is the one that counts — a per-service compose file is never allowed to shadow the actual stack definition.
At least three "real" services: a build.gradle* whose content applies the Spring Boot plugin
(org.springframework.boot/spring-boot) and whose directory has a src/main — a bare build
file with no real source tree next to it doesn't count toward the three.
A real Gradle wrapper in at least one Gradle root: a gradlew script over 1 KB and a
gradle/wrapper/gradle-wrapper.properties file next to it — a stub gradlew with no properties
file (or vice versa) fails this.
A file named openapi*/api* (.yml/.yaml/.json) that actually looks like an OpenAPI
document: it must contain both a top-level openapi: key and a top-level paths: key — a tolerant
text scan over the raw file, not a full YAML parse.
react literally appears inside some package.json's own "dependencies" block — a mention in
devDependencies, a comment, or prose doesn't count. Scored on L7_full_platform only (no smaller
rung asks for a frontend at all) — see Task Ladder.
At least one real database migration file: a Flyway db/migration/*.sql file, or any file with
changelog in its name (Liquibase), over 20 bytes.
Also offline, weight 2 each. Implemented directly in RunOracle.java. Scored only on
L7_full_platform (all three) and L3p_point_in_time (P2 only) — no other rung asks for a
written plan at all, so no other rung scores one.
docs/TASK_DEFINITION.md (≥200 chars) is checked against eight required sections — scope/overview,
actors, domain model, service boundaries, non-functional requirements, out-of-scope items, money
handling, and order states — each matched by its own heading/keyword regex. Passes when at most
one of the eight is missing, not all eight.
docs/IMPLEMENTATION_PLAN.md is parsed by the real plan parser (PlanParser — the same one RunBench
itself uses to drive an agent-authored plan), and every parsed subtask must carry a non-blank goal,
a dependency list, and a non-blank acceptance criterion. An unparseable plan is a straight FAIL,
not NOT_ATTEMPTED.
No non-doc source was written during phases 0/1. Checked from the git snapshots the runner itself
committed at the start of the run and after phase 1 (git diff --name-only start..p1, docs/task
excluded) — not a scan of the finished workspace, which every successfully implemented run would
otherwise fail. Falls back to scanning the final tree only when no snapshot pair was recorded (e.g.
scoring an imported fixture directly).
Regex-over-source checks, weight 1 each — "a cheap early signal, not the correctness oracle" (that's
the black-box F* suite below). MoneySafetyChecks.java is the implementation; every scan strips
comments first.
No money-named field, variable, or SQL column (price/amount/balance/cash/total/cost/
fee(s)/qty/quantity/notional/premium/margin) is ever declared float/double — covers
Java, Kotlin (x: Double), and SQL (FLOAT/FLOAT8/REAL/DOUBLE PRECISION column types).
Of every source file that actually handles a money-named field, at least half must import java.math.BigDecimal — a weaker bar than "every file," deliberately (a file can touch a money field
without owning the arithmetic).
No .equals() call on a money-handling value — the classic BigDecimal scale trap, where
new BigDecimal("1.0").equals(new BigDecimal("1.00")) is false despite being numerically equal.
Every point-in-time method definition found (holdingsAt/positionsAt/balanceAt/asOf/
pointInTime/snapshotAt/replay*, or any method taking an Instant-family parameter in a file
whose name hints at holdings/position/ledger) has its own method body scanned for subtraction-shaped
arithmetic on the sell side (.subtract(, -=, .negate(), a literal * -1, a signed CASE WHEN ... SELL, etc.). NOT_ATTEMPTED, not FAIL, when no such method exists at all to inspect.
Docker-gated (SKIPPED when Docker is unreachable), weight 3 each. BuildChecks.java runs every
discovered Gradle root inside a pinned gradle:8.14-jdk21 container — the agent's own wrapper is
never executed — against a real, freshly-reset PostgreSQL sidecar, on a source-only scratch copy
(build outputs, .git, node_modules, etc. stripped first; .jar/.class/.war files excluded
except the wrapper jar itself, which is source, not a build artefact).
Every discovered Gradle root runs gradle assemble and must succeed.
gradle test, verified by actually parsing the JUnit-style XML under build/test-results/ —
executed > 0 and failed == 0 across every root, not just a green exit code. A dedicated detector
also catches a neutered test task: ignoreFailures = true, enabled = false, a test class
exclusion pattern, or a bare -x test/tasks.test.enabled = false anywhere in the build script.
A seeded mutation test. One of five syntactic mutations (.subtract(→.add(, .negate()→
identity, a dropped unary minus on a qty/quantity/amount token, -=→+= on a money line,
.minus(→.plus() is applied to the real sell-side arithmetic candidate found nearest a
SELL/BUY/Side.* token or a point-in-time method definition (within 3000 characters) — and the
agent's own test suite, re-run against the mutated source, must fail. NOT_ATTEMPTED when no
such arithmetic exists anywhere in src/main to mutate in the first place — B3 never penalizes a
rung whose scenario has nothing sell-side to protect.
Also Docker-gated, against the agent's own docker compose stack, built and run from the same
source-only scratch copy. ComposeChecks.java is the implementation; two anti-gaming guards run
before any build even starts — a Dockerfile may not COPY/ADD a prebuilt artefact without
--from= (no image may depend on something a clean checkout doesn't contain), and the compose file
may not use absolute host paths for a build context or bind mount (nothing may escape the
source-only copy).
Weight 3. At least 3 non-database services report Docker healthy. A service's name is treated as
"database" by a hint list (postgres/db/database/redis/kafka/zookeeper/rabbit/mongo/
flyway/liquibase/migrat) and is never counted as a failing service even if it's still
starting slowly — a service is only "bad" if it's dead/restarting, reports unhealthy, or exited
non-zero (a one-shot migration container exiting 0 is neutral, not bad).
Weight 2. GET {base}/health returns 200 — polled up to 6 times with a 5s backoff between attempts,
against whatever base URL the run actually targets.
"THE correctness score," in BlackboxScenarios.java's own words: real HTTP scenarios the oracle
drives against the agent's live stack — the agent never sees these specific requests. Every scenario
creates its own account, so a pass on one is statistically independent of the others. Time always
comes from the server's own response (executedAt), never the oracle's clock, so clock drift can
never fail a correct implementation. F1–F5 are weight 5 except F6 (weight 4); F7–F9 are
weight 5 each.
Buy 2 TSLA @ 250.00 (balance 1000.00 → 500.00), then sell the same 2 — balance and holdings must
both land back exactly where they started. The intermediate state (after the buy, before the
sell) is checked too, not just the start and end — a no-op server that never actually debits anything
cannot pass this by accident.
Buy 5 AAPL, wait 1.5s, sell the same 5. asOf = just after the buy's own executedAt must show the
5-share position; asOf = just after the sell's own executedAt must show 0. (A replay that
adds sells instead of subtracting them would show 10 at the second point — exactly the bug M4
lints for, here caught functionally instead.)
BUY 1,000,000 AAPL on a thin balance → 422; a SELL with no prior holdings → 422. The
account's balance must be byte-for-byte unchanged across both rejected attempts.
Cancel an order that already filled → 409, and GETting the order afterward must still show
FILLED — because every order in this benchmark fills immediately, a freshly-created order is
always already terminal, so this scenario is reachable on every single run.
Deposit "0.10", then "0.20" → the resulting availableBalance must be the string "0.30"
exactly — the classic binary-floating-point 0.1+0.2 trap, caught at the API boundary.
Grades every response observed by every other F* scenario against the frozen contract's own
declared status codes and response schemas (a dedicated SCHEMAS table: which fields must be
present, and whether each is a decimal string, a timestamp, a plain string, or a map), and
separately against the agent's own shipped OpenAPI spec — a spec that doesn't declare a call the
oracle actually made and got a real response for is a finding too. This check runs last, on
purpose: it has to see the complete set of exchanges F1–F5/F7–F9 produced, including the one
POST /orders call that can ever come back 200 instead of 201 (F9's idempotent repeat) — a
response shape nothing else in the suite ever produces.
Bespoke. Two fresh accounts, one deposits "0.005", the other "0.015" — must read back exactly
"0.00" and "0.02" respectively (HALF_EVEN rounding; HALF_UP would give 0.01/0.02, naive
truncation would give 0.00/0.01 — three plausible-looking wrong answers this one scenario tells
apart).
Bespoke. Two buys of AAPL, 1.2s apart. asOf = the second buy's own executedAt must show
holdings from only the first buy (the boundary is exclusive — an order executed at exactly
asOf is not yet included), and the response's own asOfApplied field must echo that exact instant,
to better than 1ms, ending in a literal Z.
Bespoke. The same Idempotency-Key header on two identical POST /orders calls: the first returns 201 and
actually executes (balance moves from 1000.00 to 980.00); the repeat must return the same
orderId with 200 (not 201), and the account's balance and holdings afterward must show
exactly one execution, not two.
- Task Ladder — which of these 30 checks each rung actually scores
-
Features: Scoring —
CheckStatus, weights, the fixed denominator, and gaming resistance -
Architecture §6: The oracle — how
RunOraclesequences offline checks, Docker gating, and scoring