Skip to content

v0.4.0 — Portable benchmarks and community submissions

Choose a tag to compare

View all tags
@DanielTea DanielTea released this 05 Oct 06:22
v0.4.0
486b6f1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Run a reproducible benchmark and prepare a community submission with Docker:

git clone https://github.com/DanielTea/generalgamebench.git
cd generalgamebench
./ggbench

The public image contains ten scenarios: two bundled 2D games and eight ViZDoom scenarios. The launcher selects Linux AMD64 or ARM64, runs 50 fixed episodes, verifies every native replay and creates upload-ready evidence under runs/submission/. Open SUBMIT.txt and attach all ZIP files from uploads/ to the review form. No local Python, registry token, game account or model API key is needed for the bundled example policy.

Bring a custom JSONL agent with --agent-command, an immutable policy/model revision and a hardware description. Starter, portable and extended suite definitions prevent incomplete or incompatible runs being presented as comparable results. Archives record evaluator and container identities, all completed trial evidence, native images, timings and calculated rankings. Browser upload parts stay below GitHub's attachment limit and are verified when reassembled.

The verifier never executes submitted agent code. It rejects changed images or scores, missing episodes, unsafe archive paths, links, duplicates, malformed parts and false official-trust claims. Local evidence remains provisional: successful replay does not authenticate timing or grant an official rank. The default exhibition mode includes slower agents; realtime mode retains the strict sub-100 ms rule.

Validation: 85 unit checks passed. The full 50-episode portable campaign and archive replay passed independently on native Linux AMD64 and ARM64 CI runners. Docker Desktop on the Apple Silicon Mac also passed a 50-episode control campaign and a separate custom-policy submission. Windows through WSL2 is documented but has not been independently tested.

The full 43-scenario suite still needs additional runtimes, and nine catalog integrations remain unfinished. Existing model rankings are unchanged. No new hosted model calls were made. The versioned submission format is independent of the hosting platform and ready for a future Hugging Face intake service.

Pinned public image:
ghcr.io/danieltea/generalgamebench@sha256:1fabc2750c87ba08f4429715b2e7a07b36c20464d554744eba1164d9bda4eb2a

Run and submit guide · Validation records · Leaderboard

Assets 2
Loading