MapleBench v0.1.0 — a repeated Hero pilot

MapleBench v0.1.0 — a repeated Hero pilot

MapleBench lets AI agents play a real MapleStory-compatible client by writing
short programs that observe the game and press ordinary keys. This first release
compares two OpenAI and two Claude configurations on one level-180 Hero fixture,
with four predeclared attempts per model.

• Results and replay: https://maplebench.vercel.app/
• Method: docs/V1_COHORT.md at this tag.
• Machine-readable results and all fourteen original recordings:
  maplebench-v0.1.0-public-package.zip below, with its receipt. Note the unmet
  operational gate described further down.
• Sampled recording review: sampled-recording-review.json below.
• Accepted native qualification: native-qualification.json and its original video below.
• Earlier public experiments: pre-v1-public-snapshots.zip, preserved separately.

Every model gets the same frozen knowledge, 17 mapped active skills, structured
observations and a five-minute controller budget. Model wait consumes that budget.
The score is signed XP saved after ordinary logout, including its bounded tail.
Video replays show held keys and the timing of model requests and input delivery.

Results

Model	Verified	Planned	Mean net XP	Median	Min	Max	Sample SD
gpt-6-astra	3	4	32,083	32,000	27,500	36,750	4,626
gpt-5.6-sol	4	4	25,188	27,500	18,250	27,500	4,625
claude-opus-5	3	4	22,833	18,250	18,250	32,000	7,939
claude-sonnet-5	4	4	16,000	16,000	13,750	18,250	2,598

Rows keep the predeclared model order and are not sorted by score.

All sixteen dispositions

All sixteen predeclared attempts were submitted and settled; the experiment is
sealed with scope entire_declared_attempt_set. Fourteen completed with verified
persisted artifacts and published original recordings. Two are retained failures
that remain in the denominator:

• One claude-opus-5 attempt reached settlement_upload_timeout after the model
  had already acted. Its recording was discarded; its evidence was saved.
• One gpt-6-astra attempt failed earlier with adaptive_input_receipt_uncertain.

Both carry api_outcome: uncertain with the full request and token envelope
charged. Neither was retried. The runner's policy is failure: stop, retries: 0
and resume: explicit_future_unsubmitted_only, so recovery could only advance into
attempts that had never been submitted. The public disposition deliberately does not
attribute these to the model or to the infrastructure.

Operational reliability gate did not pass

The public package archive below is published with its operational reliability
gate unmet. The gate asks for three consecutive clean four-attempt groups,
where clean means all four attempts carry both a verified score and verified
recording bytes. The observed cohort produced:

Group	Attempts	Verified recordings	Clean
1	4	4	yes
2	4	2	no (recovery_invalidated x2)
3	4	4	yes
4	4	4	yes

That is two consecutive clean groups against a required three, so
gate_passed: false. The gate was not relaxed or recomputed; publishing anyway
was an explicit decision, and the unmet result travels with the artifact. The
archive's README states it, its receipt records
published_with_unmet_gate_acknowledged: true alongside the per-group counts, and
results.json inside the archive and on the site carries the same reliability
object. This gate describes observed release operations; it is not a service level
claim, and it does not affect the verification status of the fourteen scored
attempts, whose scores derive from persisted artifacts rather than recordings.

Sampled recording review

Fourteen published recordings each received a sampled visual review, with seventy
frames actually inspected rather than merely decoded. Thirteen passed. One is
flagged attention_required: two of its sampled frames render the HUD text doubled
by a capture-time compositing artifact. Its model header stayed sharp and correct,
and its score derives from persisted artifacts rather than from the recording. The
two attempts without published recordings are recorded as unavailable.

The prepared sampler targets the request end of each request-to-acknowledgment
interval, where no key is held yet, so its first input frame showed no held key in
any run. Additional candidates near the acknowledgment end, permitted by the review
policy, visually confirmed a held key in eight of the fourteen recordings. Held-key
text and highlighted controls agreed in every inspected frame. The scope is sampled
frames, not continuous playback, and not inspection of every frame.

Limits

These are descriptive results from one fixture with three or four repetitions per
model. They do not establish an overall model or provider ranking, and the
published results carry ranked: false. The native check qualifies ten core skill
effects; seven additional active controls are available but their effects remain
unqualified. Structured observations omit buff state, combo-orb count, cooldowns
and monster HP. Game RNG is uncontrolled. Trusted runtime receipts and hashes
establish consistency, not proof against a dishonest host. This is not a
vision-only benchmark.

Evidence and checks

The evaluated source revision is f3a71f5096be4651d9ce95e6436a2018678680d4;
later commits on the release branch are documentation only. Hosted CI passed on
the evaluated source, including the native runner's process-ownership test and the
full-history secret scan. The tracked-file guard and redacted history scan passed
before every push.

The companion agent-devops repository's hosted CI is blocked by account billing
and did not run. Its isolated direct-equivalent checks passed instead: 51 tests
with five expected skips, the fleet registry check, generated-artifact drift check
and ansible lint. This is not a claim that the companion's hosted CI succeeded.

The source includes the harness, verification logic, knowledge and UI. Game assets,
credentials, database contents and private runtime configuration are excluded.
Independent execution requires compatible assets and a separately provisioned
runtime. The historical snapshot asset preserves previously public bytes; it is not
renewed verification of those older experiments.