Repository navigation
Nullius v0.1.0
v0.1.0 · First release
- Spending caps. Give a recipe a budget; each unit gets its own cap, and a run stops when it is reached. Costs come from your own price list, with cache writes and one-hour retention priced at their own rates.
- Rate limits. A rate-limited run slows down and runs each rate-limited unit once more.
- Podman and more Docker setups. The container backend runs on Podman, rootless Docker, rootful Docker with userns-remap, and the Docker that
docker context useselects. - Model providers. Add several named OpenAI-compatible endpoints, and test a provider's URL and key without spending credits.
- New benchmark: GAIA on the Nullius standard. LongMemEval runs on one question type; benchmarks can grant tasks internet access.
- Safer runs. Pinned images are checked against registry tags, GPU memory is checked before dense retrievers start, and the relay's container gets its key as a file.
- More accurate results. The adjusted rate leaves out stopped trials, merged jobs replace their source shards, and runs that never began count as not run.
- Benchmark fixes. SWE-bench grades the actual patch and starts agents in the task environment; each Terminal-Bench task gets tmux; HumanEval Original setup checks run isolated.
- Interface. Long names wrap instead of being cut, and a page tells you when it is out of date.
Contributors
Thank you to everyone who built this first release:
@aitianshi, @harenome, @vultkayn, @yiannislamprou, @42int42 and @opPO333.
Install
Install the latest version with uv and Python 3.12+:
uv tool install nullius-eval
nullius serveFor an existing unpinned installation, run uv tool upgrade nullius-eval.
Open http://127.0.0.1:5057.
Full changelog
What's Changed
- fix(catalogue): pass the mem0 session date as the timestamp mem0's fix reads by @aitianshi in #29
- fix(standard): prove memory recall with a name, not a password by @aitianshi in #30
- feat(catalogue): run LongMemEval on one question type by @aitianshi in #32
- fix(standard): give drivers only the nullius package on PYTHONPATH by @aitianshi in #34
- feat(standard): price runs from the user's own price list, not a table in the code by @aitianshi in #39
- fix(standard): resolve a licence server before its forwarder takes the server's name by @harenome in #43
- fix(standard): give the relay's container its key as a file by @harenome in #44
- feat(standard): stop a run at a spending cap by @aitianshi in #46
- feat(builder): give a recipe a spending cap and show it on the job page by @aitianshi in #48
- feat(standard): give each unit a spending cap of its own by @aitianshi in #49
- fix(standard): price tokens written to a cache at the cache-write rate by @aitianshi in #51
- docs(catalogue): move an option out of the limitations and fix the inventories' pointers by @aitianshi in #57
- docs(catalogue): say the upstream defects that keep two options out by @aitianshi in #58
- feat(catalogue): name the workspace files a HumanEval solution imports and the check does not run by @aitianshi in #59
- fix(runs): count what a finished run never began as not run, not queued by @aitianshi in #60
- fix(web): wrap a sidebar name instead of cutting it, and pad the judge paragraph by @aitianshi in #61
- fix(runs): keep a workspace named after a long label listed by @aitianshi in #65
- fix(web): wrap names in the switcher, the breadcrumb and the Runs page by @aitianshi in #66
- fix(standard): stop the loopback forwarder's listener when the run stops it by @harenome in #68
- fix(standard): run the docker backend on a rootless engine by @harenome in #72
- fix(standard): isolate HumanEval Original setup checks by @aitianshi in #74
- fix(standard): reach the Docker that
docker context useselects by @harenome in #76 - fix(standard): install an agent's packages in the base image on the docker backend by @harenome in #78
- fix(standard): let uv download its Python for installs on this machine by @harenome in #80
- fix(catalogue): grade the actual SWE-bench patch by @aitianshi in #81
- fix(standard): start SWE-bench agents in the task environment by @aitianshi in #83
- fix(runs): count merged jobs in place of their source shards by @aitianshi in #84
- feat(standard): check GPU memory before starting dense retrievers by @aitianshi in #85
- fix(runs): exclude stopped trials from the adjusted rate by @aitianshi in #86
- feat(standard): check pinned images against registry tags by @aitianshi in #87
- fix(standard): separate cache writes and price one-hour retention by @aitianshi in #88
- fix(standard): prepare each Terminal-Bench task's container with tmux by @aitianshi in #89
- feat(standard): pace a run after a rate limit and run each rate-limited unit once more by @aitianshi in #90
- feat(standard): pin files for LongMemEval index expansion by @aitianshi in #91
- fix(standard): run the docker backend on a rootful Docker with userns-remap by @harenome in #93
- feat(standard): route tau3 rerankers and Qwen searches through the relay by @aitianshi in #94
- feat(standard): let benchmarks grant internet to tasks by @aitianshi in #95
- feat(standard): support Podman as the container engine by @harenome in #97
- refactor(standard): name the docker backend container, keep docker as an alias by @harenome in #99
- feat(settings): add several named OpenAI-compatible endpoints by @harenome in #101
- fix(settings): delete a host's uploaded key file when the host is removed by @harenome in #103
- feat(settings): test a model provider's URL and key without spending credits by @harenome in #105
- Add GAIA on the Nullius standard by @aitianshi in #106
- fix(web): tell the user when a page is out of date by @harenome in #111
- docs(docs): tighten the README quickstart and introduction by @aitianshi in #113
Test and CI changes: #17, #25, #35, #75, #82.
New Contributors
Full Changelog: v0.1.0rc2...v0.1.0
Installation options, downloads and validation
To pin this version, use uv tool install nullius-eval==0.1.0.
Alternatively, install the attached wheel with
uv tool install ./nullius_eval-0.1.0-py3-none-any.whl.
Add --force when replacing an existing tool installation. Settings,
catalogue additions and runs stay outside the installed package;
nullius paths shows their locations.
The downloads contain the wheel, source archive and SHA256SUMS.
After downloading all three files into one directory, verify them on Linux
with sha256sum --check SHA256SUMS.
Release validation targets Linux with Python 3.12: backend and route tests,
installed commands and resources, and user-storage preservation. It does not
establish macOS or Windows support or validate every live benchmark/provider
combination. These are Python packages with a local web UI.