Skip to content

Nullius v0.1.1

Latest

Choose a tag to compare

@github-actions github-actions released this 16 Nov 10:29
72981c9

v0.1.1 · Hugging Face endpoints, OSWorld and machine constraints

  • Hugging Face endpoint. Send a run's Hugging Face downloads to a mirror such as https://hf-mirror.com from Settings → Data providers, or with HF_ENDPOINT. It reaches drivers, Original setups, MCP servers and runs on hosts; pinned files are still checked by their sha256.
  • Machine constraints. Set the CPUs, memory, processes, concurrent units and GPUs Nullius may use on each machine. Test checks that each limit is enforced, and standard runs apply them to their containers and processes. Run hosts is now one table.
  • New benchmark: OSWorld1 on the Nullius standard: OSWorld's Ubuntu tasks, each in its own VM, with OSWorld's own agents as the Original setup and a null control. The standard defines the agent.computer/v1 port it declares, and a locked install fetches the git submodules its lock names.
  • GSM8K-Platinum. GSM8K offers the cleaned test split as a second dataset, with the same unit numbers as the original.
  • Run listed units. nullius standard run ... --units FILE runs only the units a file names; merge refuses shards whose lists differ.
  • Runs by launch. A participant run over several shards is one row with the merged numbers, and unfolds into one row per shard. Controls are drawn whatever their state, and the task-check line names a control that did not check every task.
  • Fix. A model provider's Remove appears and disappears after a save without a reload.

Contributors

Thank you to everyone who built this release:
@aitianshi, @harenome and @pierreleca.

Install

Install the latest version with uv and Python 3.12+:

uv tool install nullius-eval
nullius serve

For an existing unpinned installation, run uv tool upgrade nullius-eval.
Open http://127.0.0.1:5057.

Full changelog

What's Changed

  • feat(standard): define agent.computer/v1 and install a locked project's git submodules by @pierreleca in #114
  • feat(catalogue): add OSWorld1 with OSWorld's own agents and a null control by @pierreleca in #117
  • feat(standard): run the units a file lists with --units by @pierreleca in #120
  • feat(settings): set and enforce what Nullius may use on each machine by @harenome in #122
  • fix(settings): bring a provider card's Remove up to date after a save by @harenome in #125
  • fix(runs): draw a launch's controls whatever their state by @harenome in #127
  • feat(settings): send Hugging Face downloads to a chosen endpoint by @aitianshi in #129
  • feat(catalogue): offer GSM8K-Platinum as a second dataset of GSM8K by @aitianshi in #132
  • feat(runs): fold a launch's shards into one row per participant by @harenome in #137

Test and CI changes: #135.

New Contributors

Full Changelog: v0.1.0...v0.1.1

Installation options, downloads and validation

To pin this version, use uv tool install nullius-eval==0.1.1.
Alternatively, install the attached wheel with
uv tool install ./nullius_eval-0.1.1-py3-none-any.whl.
Add --force when replacing an existing tool installation. Settings,
catalogue additions and runs stay outside the installed package;
nullius paths shows their locations.

The downloads contain the wheel, source archive and SHA256SUMS.
After downloading all three files into one directory, verify them on Linux
with sha256sum --check SHA256SUMS.

Release validation targets Linux with Python 3.12: backend and route tests,
installed commands and resources, and user-storage preservation. It does not
establish macOS or Windows support or validate every live benchmark/provider
combination. These are Python packages with a local web UI.