Skip to content

Nullius v0.1.0

Choose a tag to compare

@github-actions github-actions released this 13 Nov 10:47
a7374d7

v0.1.0 · First release

  • Spending caps. Give a recipe a budget; each unit gets its own cap, and a run stops when it is reached. Costs come from your own price list, with cache writes and one-hour retention priced at their own rates.
  • Rate limits. A rate-limited run slows down and runs each rate-limited unit once more.
  • Podman and more Docker setups. The container backend runs on Podman, rootless Docker, rootful Docker with userns-remap, and the Docker that docker context use selects.
  • Model providers. Add several named OpenAI-compatible endpoints, and test a provider's URL and key without spending credits.
  • New benchmark: GAIA on the Nullius standard. LongMemEval runs on one question type; benchmarks can grant tasks internet access.
  • Safer runs. Pinned images are checked against registry tags, GPU memory is checked before dense retrievers start, and the relay's container gets its key as a file.
  • More accurate results. The adjusted rate leaves out stopped trials, merged jobs replace their source shards, and runs that never began count as not run.
  • Benchmark fixes. SWE-bench grades the actual patch and starts agents in the task environment; each Terminal-Bench task gets tmux; HumanEval Original setup checks run isolated.
  • Interface. Long names wrap instead of being cut, and a page tells you when it is out of date.

Contributors

Thank you to everyone who built this first release:
@aitianshi, @harenome, @vultkayn, @yiannislamprou, @42int42 and @opPO333.

Install

Install the latest version with uv and Python 3.12+:

uv tool install nullius-eval
nullius serve

For an existing unpinned installation, run uv tool upgrade nullius-eval.
Open http://127.0.0.1:5057.

Full changelog

What's Changed

  • fix(catalogue): pass the mem0 session date as the timestamp mem0's fix reads by @aitianshi in #29
  • fix(standard): prove memory recall with a name, not a password by @aitianshi in #30
  • feat(catalogue): run LongMemEval on one question type by @aitianshi in #32
  • fix(standard): give drivers only the nullius package on PYTHONPATH by @aitianshi in #34
  • feat(standard): price runs from the user's own price list, not a table in the code by @aitianshi in #39
  • fix(standard): resolve a licence server before its forwarder takes the server's name by @harenome in #43
  • fix(standard): give the relay's container its key as a file by @harenome in #44
  • feat(standard): stop a run at a spending cap by @aitianshi in #46
  • feat(builder): give a recipe a spending cap and show it on the job page by @aitianshi in #48
  • feat(standard): give each unit a spending cap of its own by @aitianshi in #49
  • fix(standard): price tokens written to a cache at the cache-write rate by @aitianshi in #51
  • docs(catalogue): move an option out of the limitations and fix the inventories' pointers by @aitianshi in #57
  • docs(catalogue): say the upstream defects that keep two options out by @aitianshi in #58
  • feat(catalogue): name the workspace files a HumanEval solution imports and the check does not run by @aitianshi in #59
  • fix(runs): count what a finished run never began as not run, not queued by @aitianshi in #60
  • fix(web): wrap a sidebar name instead of cutting it, and pad the judge paragraph by @aitianshi in #61
  • fix(runs): keep a workspace named after a long label listed by @aitianshi in #65
  • fix(web): wrap names in the switcher, the breadcrumb and the Runs page by @aitianshi in #66
  • fix(standard): stop the loopback forwarder's listener when the run stops it by @harenome in #68
  • fix(standard): run the docker backend on a rootless engine by @harenome in #72
  • fix(standard): isolate HumanEval Original setup checks by @aitianshi in #74
  • fix(standard): reach the Docker that docker context use selects by @harenome in #76
  • fix(standard): install an agent's packages in the base image on the docker backend by @harenome in #78
  • fix(standard): let uv download its Python for installs on this machine by @harenome in #80
  • fix(catalogue): grade the actual SWE-bench patch by @aitianshi in #81
  • fix(standard): start SWE-bench agents in the task environment by @aitianshi in #83
  • fix(runs): count merged jobs in place of their source shards by @aitianshi in #84
  • feat(standard): check GPU memory before starting dense retrievers by @aitianshi in #85
  • fix(runs): exclude stopped trials from the adjusted rate by @aitianshi in #86
  • feat(standard): check pinned images against registry tags by @aitianshi in #87
  • fix(standard): separate cache writes and price one-hour retention by @aitianshi in #88
  • fix(standard): prepare each Terminal-Bench task's container with tmux by @aitianshi in #89
  • feat(standard): pace a run after a rate limit and run each rate-limited unit once more by @aitianshi in #90
  • feat(standard): pin files for LongMemEval index expansion by @aitianshi in #91
  • fix(standard): run the docker backend on a rootful Docker with userns-remap by @harenome in #93
  • feat(standard): route tau3 rerankers and Qwen searches through the relay by @aitianshi in #94
  • feat(standard): let benchmarks grant internet to tasks by @aitianshi in #95
  • feat(standard): support Podman as the container engine by @harenome in #97
  • refactor(standard): name the docker backend container, keep docker as an alias by @harenome in #99
  • feat(settings): add several named OpenAI-compatible endpoints by @harenome in #101
  • fix(settings): delete a host's uploaded key file when the host is removed by @harenome in #103
  • feat(settings): test a model provider's URL and key without spending credits by @harenome in #105
  • Add GAIA on the Nullius standard by @aitianshi in #106
  • fix(web): tell the user when a page is out of date by @harenome in #111
  • docs(docs): tighten the README quickstart and introduction by @aitianshi in #113

Test and CI changes: #17, #25, #35, #75, #82.

New Contributors

Full Changelog: v0.1.0rc2...v0.1.0

Installation options, downloads and validation

To pin this version, use uv tool install nullius-eval==0.1.0.
Alternatively, install the attached wheel with
uv tool install ./nullius_eval-0.1.0-py3-none-any.whl.
Add --force when replacing an existing tool installation. Settings,
catalogue additions and runs stay outside the installed package;
nullius paths shows their locations.

The downloads contain the wheel, source archive and SHA256SUMS.
After downloading all three files into one directory, verify them on Linux
with sha256sum --check SHA256SUMS.

Release validation targets Linux with Python 3.12: backend and route tests,
installed commands and resources, and user-storage preservation. It does not
establish macOS or Windows support or validate every live benchmark/provider
combination. These are Python packages with a local web UI.