Skip to content

v0.3.1: verified numbers

Latest

Choose a tag to compare

@hbofz hbofz released this 01 Sep 19:40
· 7 commits to main since this release

No emulator or kernel changes. This release is about making the project's claims
checkable and the repository contributable.

Every published number re-measured

All five A100 throughput claims were re-measured from a clean clone and
reproduced at or above their published values. Two independent runs, on a
40 GB and an 80 GB A100, agreed to within 0.3%.

Envs Previously published Measured Verdict
4,096 311,844 310,903 PASS
16,384 1,050,710 1,108,937 PASS
32,768 1,931,460 2,004,806 PASS
65,536 3,142,205 3,267,050 PASS
131,072 2,816,144 3,055,618 PASS

69/69 tests pass and all three falsifiability checks pass on the same run. Raw
output is committed at docs/data/verification-2026-09-01-a100.json.

Measured for the first time:

  • Crossover at 8 environments, not the 32 and 64 the docs claimed. Both
    figures predated the 2026-07-29 optimization program.
  • Device memory of ~195 KB per environment, 12.2 GiB at the 65,536-env
    peak. This confirms a "13 GB" figure the README had asserted with no
    measurement behind it.
  • Linear scaling holds to ~8,192 envs, not ~2,048 as documented.

Reproduce it yourself

benchmarks/verify_claims.py reuses the functions in benchmarks/gpu_vs_cpu.py,
so the protocol matches the published tables exactly. It prints a
claimed-vs-measured table and exits non-zero if any claim, test, or check fails.
Run it directly, through notebooks/verify_claims.ipynb on Colab, or on a
rented GPU via the Colab CLI.

Corrected

Stale numbers were traced to the commit that wrote them and fixed:

  • The nes-py comparison read 269x, computed against an intermediate build. It
    is ~1,370x.
  • docs/design.md quoted throughput 6x below the README's own headline, with
    no indication it was historical.
  • docs/benchmark-gpu-vs-cpu.md predates the entire optimization program and
    now carries a banner naming the specific corrections.
  • KNOWN_ISSUES.md carried superseded A100 figures and claimed the C++ tests
    were not in CI; they have been since 394d090.
  • MANIFEST.in omitted scripts/*.py, so sdists shipped without the build
    script the README tells users to run.
  • A vendored doc cloned a repository name that no longer exists.

Contributing

  • CONTRIBUTING.md: setup, the checks to run, and the bar for emulator and
    performance changes.
  • Nine issues filed and labelled, each pointing at a specific file. The largest
    is mapper support beyond NROM (#5), which would take NeSLE from a Mario
    trainer to a general NES RL platform.
  • docs/install.md splits the platform-specific build detail out of the README.
  • Discussions enabled.