Repository navigation
v0.1.1 · Hugging Face endpoints, OSWorld and machine constraints
- Hugging Face endpoint. Send a run's Hugging Face downloads to a mirror such as
https://hf-mirror.comfrom Settings → Data providers, or withHF_ENDPOINT. It reaches drivers, Original setups, MCP servers and runs on hosts; pinned files are still checked by their sha256. - Machine constraints. Set the CPUs, memory, processes, concurrent units and GPUs Nullius may use on each machine. Test checks that each limit is enforced, and standard runs apply them to their containers and processes. Run hosts is now one table.
- New benchmark: OSWorld1 on the Nullius standard: OSWorld's Ubuntu tasks, each in its own VM, with OSWorld's own agents as the Original setup and a null control. The standard defines the
agent.computer/v1port it declares, and a locked install fetches the git submodules its lock names. - GSM8K-Platinum. GSM8K offers the cleaned test split as a second dataset, with the same unit numbers as the original.
- Run listed units.
nullius standard run ... --units FILEruns only the units a file names; merge refuses shards whose lists differ. - Runs by launch. A participant run over several shards is one row with the merged numbers, and unfolds into one row per shard. Controls are drawn whatever their state, and the task-check line names a control that did not check every task.
- Fix. A model provider's Remove appears and disappears after a save without a reload.
Contributors
Thank you to everyone who built this release:
@aitianshi, @harenome and @pierreleca.
Install
Install the latest version with uv and Python 3.12+:
uv tool install nullius-eval
nullius serveFor an existing unpinned installation, run uv tool upgrade nullius-eval.
Open http://127.0.0.1:5057.
Full changelog
What's Changed
- feat(standard): define agent.computer/v1 and install a locked project's git submodules by @pierreleca in #114
- feat(catalogue): add OSWorld1 with OSWorld's own agents and a null control by @pierreleca in #117
- feat(standard): run the units a file lists with --units by @pierreleca in #120
- feat(settings): set and enforce what Nullius may use on each machine by @harenome in #122
- fix(settings): bring a provider card's Remove up to date after a save by @harenome in #125
- fix(runs): draw a launch's controls whatever their state by @harenome in #127
- feat(settings): send Hugging Face downloads to a chosen endpoint by @aitianshi in #129
- feat(catalogue): offer GSM8K-Platinum as a second dataset of GSM8K by @aitianshi in #132
- feat(runs): fold a launch's shards into one row per participant by @harenome in #137
Test and CI changes: #135.
New Contributors
- @pierreleca made their first contribution in #114
Full Changelog: v0.1.0...v0.1.1
Installation options, downloads and validation
To pin this version, use uv tool install nullius-eval==0.1.1.
Alternatively, install the attached wheel with
uv tool install ./nullius_eval-0.1.1-py3-none-any.whl.
Add --force when replacing an existing tool installation. Settings,
catalogue additions and runs stay outside the installed package;
nullius paths shows their locations.
The downloads contain the wheel, source archive and SHA256SUMS.
After downloading all three files into one directory, verify them on Linux
with sha256sum --check SHA256SUMS.
Release validation targets Linux with Python 3.12: backend and route tests,
installed commands and resources, and user-storage preservation. It does not
establish macOS or Windows support or validate every live benchmark/provider
combination. These are Python packages with a local web UI.