v0.4.0 — Portable benchmarks and community submissions

Run a reproducible benchmark and prepare a community submission with Docker:

git clone https://github.com/DanielTea/generalgamebench.git
cd generalgamebench
./ggbench

The public image contains ten scenarios: two bundled 2D games and eight ViZDoom scenarios. The launcher selects Linux AMD64 or ARM64, runs 50 fixed episodes, verifies every native replay and creates upload-ready evidence under runs/submission/. Open SUBMIT.txt and attach all ZIP files from uploads/ to the review form. No local Python, registry token, game account or model API key is needed for the bundled example policy.

Bring a custom JSONL agent with --agent-command, an immutable policy/model revision and a hardware description. Starter, portable and extended suite definitions prevent incomplete or incompatible runs being presented as comparable results. Archives record evaluator and container identities, all completed trial evidence, native images, timings and calculated rankings. Browser upload parts stay below GitHub's attachment limit and are verified when reassembled.

The verifier never executes submitted agent code. It rejects changed images or scores, missing episodes, unsafe archive paths, links, duplicates, malformed parts and false official-trust claims. Local evidence remains provisional: successful replay does not authenticate timing or grant an official rank. The default exhibition mode includes slower agents; realtime mode retains the strict sub-100 ms rule.

Validation: 85 unit checks passed. The full 50-episode portable campaign and archive replay passed independently on native Linux AMD64 and ARM64 CI runners. Docker Desktop on the Apple Silicon Mac also passed a 50-episode control campaign and a separate custom-policy submission. Windows through WSL2 is documented but has not been independently tested.

The full 43-scenario suite still needs additional runtimes, and nine catalog integrations remain unfinished. Existing model rankings are unchanged. No new hosted model calls were made. The versioned submission format is independent of the hosting platform and ready for a future Hugging Face intake service.

Pinned public image:
ghcr.io/danieltea/generalgamebench@sha256:1fabc2750c87ba08f4429715b2e7a07b36c20464d554744eba1164d9bda4eb2a

Run and submit guide · Validation records · Leaderboard

Hosted-model refresh — 5 October 2026

Eleven configured models completed the 43-scenario hosted-model refresh: seven OpenAI models and four Claude models, including Sonnet 5.5. Opus 5.5 was inaccessible through the configured connection; its failed attempt is retained without an aggregate rank.

• 645 scored episodes and 4,933 decisions, plus one unavailable-model attempt. Four reference policies use the same suite, seed and horizon.
• All 646 included records passed exact game replay before publication. 52 scored episodes aborted on provider or JSON-format failures and remain zero; no successful retries replace them.
• Airstriker now renders off-screen. Its complete 15-episode cohort replays with identical frames and scores, and a real-engine regression rejects attempts to open a display.
• The previous 33-scenario exhibition remains separate, including its local vision models. Hugging Face exports preserve the old and current cohorts separately; no Hub upload is performed.

Scoring is the equally weighted mean of normalized scenario scores, multiplied by 100. The strict under-100-ms gate is separate. None of the current rows passes it; the official track remains empty. One seed and eight decisions per game make this an integration demonstration, not a reliable comparison of general intelligence.

Verify SHA256SUMS before extracting the evidence. The archive contains selected event ledgers, result JSON and PNG observations only. See results/PROVENANCE.md and results/refresh-2026-10-05 for source hashes, exact model IDs, failures, checks and scoring details.

Validation: 88 automated tests; native Docker validation on AMD64 and ARM64; two targeted real-engine Airstriker checks; full 646-record replay audit.

Verified hosted model reruns — 5 October 2026

Eleven OpenAI and Anthropic models completed fresh runs with the current renderer. The Mac ran all 43 scenarios for each model. Hosted providers ran model inference.

The runs use seed 3000, evaluator 0.5.0, the same prompt and action protocol, and a limit of eight decisions per scenario. The Mac ran one model and one game at a time.

The new evidence contains 473 episodes, 3,721 decisions, and 11 reply errors. Every new episode passed native replay twice. All 104 fixed source checksums match. No failed episode was replaced.

The combined table keeps the four verified local vision model rows unchanged. It contains 15 models, 645 episodes, 4,208 decisions, and 138 reply errors. The earlier archives remain unchanged.

No model passed suite p95 below 200 ms. Exactly 200 ms fails. The test pools all recorded response times and uses linear interpolation. Scores, errors, and trust remain separate. All entries remain local-unattested. These short tests check integration. They do not establish model skill.

Hosted timing includes command startup, image transfer, network time, inference, and action validation. Local MLX models retain their weights in memory. The Dataset keeps these conditions under two suite identifiers. These timings do not measure model-only speed.

The evidence archive contains ledgers, saved results, and PNG observations for all 645 episodes. The report archive contains both model campaigns, the Linux validation report, and the benchmark declarations. SHA256SUMS records both archive checksums.

• Leaderboard
• Results Dataset
• Campaign report
• Fixed evaluator source
• Previous results archive

Mac vision model results and full Linux suite support

All 43 scenarios now pass native recording and exact replay on Linux x86_64. The model campaign in this release ran entirely on the Mac. Hugging Face hosts the result page and Dataset.

The new model table contains four models with pinned Hugging Face weights. Each model ran all 43 scenarios with seed 3000 and a limit of eight decisions. All 172 episodes passed native replay twice. The evidence retains 487 decisions and all 127 reply errors. None passed the suite p95 limit below 200 ms.

Model	Score / 100	Suite p95	Reply errors
LFM2.5-VL 450M, 4-bit	12.718	663.88 ms	1
SmolVLM2 256M	0.211	1565.56 ms	41
FastVLM 0.5B, BF16	0.000	1429.31 ms	42
SmolVLM2 500M	0.000	1982.70 ms	43

These short runs check integration. They do not establish model skill. All results remain local-unattested. The official leaderboard has no entries.

The evidence archive contains the saved game images, event records, and result files. The results archive contains the campaign declaration, model inventory, replay checks, Linux validation report, and previous snapshot. SHA256SUMS gives the archive checksums. Earlier releases remain unchanged.

Leaderboard · Campaign report · Fixed test source · Linux validation

The Gaming category request remains open with the Leaderboard Finder maintainers. Publication of this Space does not confirm Finder inclusion.

Complete headless baseline rerun — 5 October 2026

All four reference policies ran the complete headless suite: 43 scenarios, three seeds, 24-decision horizon, 516 episodes and 11,641 decisions. All episodes passed exact replay. No model API calls were made.

Scores / 100: Pixel Tracker 16.070; Pixel React 15.906; Random 13.325; Idle 11.727. All four fail the strict 100 ms gate. All 21 timeout episodes score zero. Results are local-unattested.

The evidence archive contains all original PNG observations, ledgers and result records. Check SHA256SUMS, extract into a fresh directory, and replay with the pinned game runtime.

A later request removed baseline and previous-result views from the public page and Hugging Face. This release preserves that layout. The fresh baseline data remains in the GitHub result archive and website data download. Existing model scores are unchanged. No Hub publication was made by this refresh.

See run provenance. No credentials, game binaries or model weights are included. Game media retains its separate rights.

v0.3.0 — Ten verified native game integrations

Ten additional native game tasks now pass admission: Unity VisualFoodCollector, Google Research Football Academy, SuperTuxKart Lighthouse, Luanti ChopTree, SuperTux, Dungeon Crawl Stone Soup, OpenTTD, Mindustry, Cataclysm: DDA and Warzone 2100. The catalog now covers 43 runnable scenarios across 36 of 45 cards.

The fixed integration-control cohort contains 21 episodes and 4,303 decisions, with zero errors or aborted episodes. Every episode passed exact independent camera, score and termination replay. Pinned builds, derivative source patches, upstream licence notices and the native admission tests are included. The final unit suite passed 68 checks, including five compact-archive checks; optional native checks run separately.

The existing v0.2 leaderboard remains unchanged at 33 scenarios. No hosted or local model was called for this validation cohort. These are local, unattested measurements; the official track remains empty.

Nine integrations remain unfinished. Native 0 A.D., StarCraft II and Veloren prototypes have reproducible diagnostics but fail visual replay or task admission. Six other game families need usable installations/assets and adapter work. Their previews and failed diagnostics never enter ranked evidence.

Assets:

• Compact evidence archive: the 21 verified control episodes, event ledgers and native PNG observations, restored byte-for-byte with the pinned evidence-codec recipe.
• Diagnostic-media archive: explicitly unranked native previews for the three prototypes.
• Hugging Face export: portable static Space and tabular Dataset, with unchanged v0.2 ranking rows and updated catalog. Nothing uploaded to Hugging Face.
• SHA256SUMS: checksums for all archives.

Restore the compact evidence with environments/evidence-codec/README.md; extract the diagnostic-media archive normally. Keep the recorded episode paths under the repository root to reproduce the card media. See results/PROVENANCE.md and docs/MEDIA.md for exact scope and instructions. Game binaries, commercial assets, model weights and credentials are not included.

GeneralGameBench 0.2.0 — 33 scenarios and 17-model exhibition

GeneralGameBench now covers 33 validated scenarios across 26 of the 45 environment cards. This release adds the 16 Procgen games, Crafter, MiniWorld, Pistonball, Breakout, Airstriker, NetHack and MiniHack.

The new snapshot contains 1,089 actual episodes and 14,394 decisions from seven configured OpenAI models, three Anthropic models, seven local vision models and four built-in reference policies. All selected episodes passed exact gameplay replay. The 44 aborted episodes remain in the results with zero scores.

Model evaluations use one seed and eight decisions per game. They demonstrate working integrations, not reliable intelligence rankings. No model met the strict under-100 ms requirement. All results remain local and unattested; official rankings are empty. Nineteen environment cards, including modern commercial AAA titles, still require implementation or validation.

Crafter task version 2 stabilizes creature-selection order across processes. Its entire cohort was regenerated, and both source revisions and the selection rule are recorded in the campaign provenance.

Validation: 83 tests passed locally, including 26 real-engine checks; GitHub checks passed; all 1,089 episodes replayed; browser controls and the production website build passed. The portable Dataset loaded successfully with Hugging Face Datasets.

Assets:

• generalgamebench-evidence-v0.2.0.tar.gz: 16,572 selected event, result and observation files. Install the pinned optional runtimes before replaying.
• generalgamebench-huggingface-v0.2.0.tar.gz: a static Space and separate Dataset, ready for future publication. No Hugging Face upload has occurred. Game media retains its separately credited rights; model weights, game binaries and credentials are excluded.
• SHA256SUMS: checksums for both archives.

Leaderboard · Blog · Provenance · Coverage

GeneralGameBench v0.1.1

GeneralGameBench is the new name for ScreenQuest Arena.

• Renamed the repository, project board, leaderboard and Python package.
• Added the generalgamebench command; arena remains available as a compatibility alias.
• Updated setup instructions and project links. The motto is unchanged.
• Preserved Season 0 results, evidence, and the screenquest/1 protocol identifier. The original source hashes and complete evidence archive remain in v0.1.0.

Validation: 49 passing tests, formatting and lint checks, package builds, and replay of an existing 80-decision episode. This release changes naming; it adds no new benchmark measurements or official rankings.

ScreenQuest Arena v0.1.0 — measured game-agent leaderboard

ScreenQuest Arena v0.1.0 is an open-source reference implementation for pixels-only game-agent evaluation.

Other benchmarks test productivity/usefulness, we test intelligence.

• Ten runnable scenarios: two original 2D games and eight ViZDoom scenarios.
• 400 measured baseline episodes, plus 36 exhibition episodes comparing actual GPT-6 Astra, Claude Opus 5 and local reference policies.
• A strict per-decision <100 ms gate; slower model runs stay in exhibition.
• Complete screenshot/action evidence, deterministic replay, fixed-suite scores and bootstrap intervals.
• 49 passing local tests and automated Linux checks.
• A public leaderboard, participant protocol, submission form and 42 researched game candidates/families.

All results are local-unattested. The model exhibition uses two seeds and an eight-decision horizon; it is an integration smoke test, not a robust model ranking. The simulation is lockstep, not continuous realtime. Modern AAA adapters and independently isolated official evaluation remain roadmap work.

The 204 MiB evidence archive includes all final-campaign observations and traces. Verify it using SHA256SUMS and the instructions in results/PROVENANCE.md. Third-party rendered game content retains its own rights; the MIT license covers original code.