v0.37.3 — the arena can read the whole submission

Completes the /sol permission fix from 0.37.1: the arena now makes the entire solution tree readable by its non-root sandbox uid (not just the top directory), so verification and evolve/eval work on native Linux even under a restrictive umask (e.g. a domain-joined host, umask 077), where the cloned files would otherwise be owner-only and every episode failed at turn 0. Host-side change; sandbox re-pinned at the same scoring major.

v0.37.2 — the broker forwards to its provider only

What's Changed

• fix(broker): refuse a request-target that re-homes the upstream (for 0.37.2) by @vkurenkov in #107
• docs: every page rewritten and verified against the code (for 0.37.2) by @vkurenkov in #108
• Release 0.37.2: the broker forwards to its provider only by @vkurenkov in #109

Full Changelog: v0.37.1...v0.37.2

v0.37.1 — the verifier can read the solution again

Fixes a permission bug that made the arena refuse to run a submission on native Linux in two cases: the hidden-seed verifier (which clones each candidate into a private temp dir), and users running evolve/eval with a restrictive umask (077). The arena runs submissions as a non-root user and couldn't enter a 0700 solution directory, so every episode failed at turn 0. eval_batch now makes the solution mount traversable. Default-umask users and Docker Desktop were unaffected.

Sandbox images re-pinned at the same scoring major — verified leaderboard unaffected.

v0.37.0 — the credential never enters the sandbox

The credential broker is now on by default for every operator (Claude, Codex) on every platform. The model credential stays host-side and is injected on the wire — the untrusted sandbox never sees it. Log into Claude once and never touch a token again (host-side auto-refresh); Codex's curl_cffi TLS helper installs itself with zero user step.

Fixes that only bit native Linux (Docker Desktop had masked them on macOS): the broker reaches the host over the docker bridge gateway, nethackers setup prints the one-time port-scoped ufw rule, and the arena output dir is writable by the sandbox — the last one had been crashing every native-Linux evolve/eval.

Host-side only: no sandbox-image behaviour change, arena re-pinned at the same scoring major (verified leaderboard unaffected).

Verified end-to-end on macOS and a real Linux host — nethackers setup, the live broker round-trip for both operators against this release's sandbox image, and a full evolve pipeline.

v0.36.2 — the sandbox knows this month's models

A coding-agent CLI resolves --model against its own allowlist, not your account's. The sandbox shipped claude 2.1.270, which answers a newer model id with [claude-code:unrecognized_model] and exits in under a second, having spent nothing — so a run started with such a model failed every attempt while the monitor said only "agent failed".

• The sandbox CLIs are current: claude 2.1.270 → 2.1.280, codex 0.154.0 → 0.156.1. OpenCode stays at 1.18.31.
• A rejected model names itself and the way out. The failure detail used to be the last line the CLI printed, which is its usage JSON — several hundred characters saying nothing. It now reads "unrecognized model 'X' — the sandbox's CLI doesn't know that id; use an alias like opus, or update the CLI in the mutator image", in the monitor, in report() and in metrics.jsonl.
• It fails once. A rejected request cannot come good on a retry, so the run stops on the first one instead of spending three iterations and two backoff waits on the same answer.

Worth knowing when you pick a model: the aliases (opus, sonnet, haiku, fable, default) are resolved by the CLI itself, so they can never go stale against the pinned image. A dated or versioned id can.

Both sandbox images are re-pinned; the arena rebuild is classified at major 2 — no scoring-path change, so the verified corpus carries over.

v0.36.1 — the monitor and the brief stop rewarding a tie

Two fixes, both the same shape: a threshold that was easier than the rule it stood for.

The monitor marked wins the loop had rejected. Progress showed "▲ new best" on iterations the iteration list called "no gain". Run.incumbent averaged the incumbent with sum()/len(); story.this_cell averaged the candidate with statistics.mean. On real scores those land one ULP apart — 0.062096559999999995 against 0.06209656 — and because a bot is kept only when it strictly beats the cell, a rejected iteration ties its incumbent exactly, which is the one place an ULP decides the answer. Every score the monitor compares now goes through a single progress_mean, the same arithmetic the harness scores candidates with.

The brief printed its bar rounded down. target to beat: {target:.3f} turned a best-so-far of 0.0672706… into "0.067", so a bot that reproduced that exact program reported "beating 0.067" — true of the number it was shown, false of the rule — and stopped there. Bars now round up: clearing what is printed always clears what is enforced.

A cell could hold a bot another elite dominates. A change that helps some identities and ties the rest left the tied cells holding the older, strictly worse bot, and those cells hand out parents — so the mutator kept being given a tree behind the archive, whose one visible route to the target was a change already banked elsewhere. A tied cell now goes to a program that strictly improves somewhere else. The tie is not reported as an improvement; only the occupant changes.

And the brief asks for more. It now states the bar it enforces and asks for a point beyond it: a change that moves the average by less than 0.01 is not worth making. The enforced rule is unchanged — anything strictly better is still kept, because a cell whose elite never moves is what caused the rediscovery in the first place.

Sandbox images are re-pinned; the arena rebuild is classified at major 2 — no scoring-path change, so the verified corpus carries over.

v0.36.0 — the evolve UX says what's happening

Three things about running nethackers evolve were measured and fixed.

Switching operators shows its check. Picking a different coding agent re-probes it inside the sandbox — 2.5s for claude, 3.4s for codex, 9.4s for opencode2 — and used to do that in total silence, with the old version line still on screen. It now says ⠹ checking codex in the sandbox… Ns while the model and effort pickers show as loading, and a probe that fails clears them.

Start answers immediately. It used to freeze the UI for 4 seconds typically and 13 at worst. The button now reads Starting… while a step line ticks ✓ Docker · ✓ sandbox · ⠹ preparing run…, every check runs off the UI thread, and a failure keeps you on the form instead of half-opening a monitor. The operator version the form already probed is reused, so Start no longer re-runs two emulated containers to learn it again.

The monitor says what is happening. It opens on Logs, a step list per iteration in plain words, under a line that always says what the run is doing right now. The iteration list names outcomes — improved, no gain, failed test, agent failed, error — and follows the live iteration until you pick another. After the first finished iteration you get · at this pace ~Xm left, computed from this run's own timings. Progress says what each identity is waiting on (waiting for the agent…, smoke test…, 0.23 11/15 games ▲ new best), explains its marks in a legend, and shows BEST OVERALL's candidate so the "or the best average" half of the keep rule is visible. The per-seed detail reads avg, not x̄.

The wording is held to what the run actually did: an errored iteration never reads as a win, a crash reads as a failure rather than as your Stop, a batch cut short shows the games it really played, and the keep-rule bar shown is the one the archive uses. A gh that does not answer within 10 seconds now has its own state instead of being reported as "not logged in", which pointed at the wrong fix.

Also: logging in from inside the TUI now reaches Start (it used to keep the credentials it was built with, so the next run went out offline), and the iterations field defaults to 100.

Sandbox images are re-pinned; the arena rebuild is classified at major 2 — no scoring-path change, so the verified corpus carries over.

v0.35.0 — one command from install to ready

Installing by hand meant a container runtime, Rosetta on Apple silicon, gh, a coding agent CLI, three logins, and a gigabyte of images fetched by a flag most people never found. nethackers setup does all of it in one command.

Setup

• One command, one question. nethackers setup checks the machine, shows you the plan, and asks once. Then it runs the logins first and the slow work last, so you can walk away after the last browser tab.
• It never runs sudo. Setup runs an install only when it is one documented command that needs no root: Homebrew on macOS, and the vendors' own installers for Claude Code and Codex. Anything needing root, a GUI click or a re-login is printed for you to run. A test enforces it.
• Every step says whether it has been tested. Each OS has its own file of recipes, and each recipe carries tested, untested or not covered with its evidence. The plan shows "(untested)" next to those commands, and docs/setup.md has the whole table. Most recipes ship as untested: they come from vendor docs and haven't been run on a real machine yet.
• Apple silicon. Setup starts Colima with Rosetta and sizes it to the machine. Without Rosetta 2, it prints how to install it first rather than starting a VM that would stall. Docker Desktop, OrbStack and Podman are detected and started, not replaced.
• Scopes. --for eval, --for evolve or --for publish set up only what that command needs.
• Coding agents. With no terminal, setup changes nothing and prints its plan, so an agent can summarize it and ask you one yes/no question. --yes --operator <claude|codex|opencode2> runs the unattended steps and lists the logins it can't run itself.

Downloads

• The image pull shows bytes, speed and time left, not a wall of layer IDs, so a slow network no longer looks like a hang. eval's first pull uses the same bar. The plan says "up to N MB", because an update downloads only what changed.

Smaller things

• doctor's fixes now say nethackers setup wherever setup can fix the check, and doctor itself stays read-only.
• The login hint said claude, which isn't a login command. It now says claude auth login.
• The Colima hint used to create a VM without Rosetta, and the Rosetta hint named a --vz flag Colima doesn't have.
• Troubleshooting starts with setup, and its Podman section no longer lists a fixed issue as open.

Under the hood

• Arena rebuilt only because the version bump moves uv.lock, and classified at ARENA_MAJOR=2: nothing on the scoring path changed, so the verified corpus carries over.

v0.34.4 — publishing never waits on a git login

An evolve run no longer stalls when git has no login for GitHub. Publishing a candidate used to push with git's own credential setup instead of your gh login; where that had nothing for github.com, git asked for a username in the terminal and the run waited, six and a half hours on one overnight run.

Publishing

• Pushes log in through gh. The push to your nethacker repo uses the same gh login as every other publish step, so you no longer need gh auth setup-git or a git credential stored for github.com.
• Never waits on a prompt. If a login is missing, the publish step fails at once and the run carries on, keeping the win local.

Under the hood

• Arena rebuilt only because the version bump moves uv.lock, and classified at ARENA_MAJOR=2: nothing on the scoring path changed, so the verified corpus carries over.

v0.34.3 — works behind TLS-inspecting firewalls

nethackers now works on networks that inspect HTTPS. A corporate firewall or antivirus that re-signs every certificate with its own CA broke every hub call: the browser, gh and git trusted that CA through the OS certificate store, but nethackers checked certificates only against its bundled list. An evolve run on such a network played for hours while every win stayed local-only with CERTIFICATE_VERIFY_FAILED.

Networking

• The OS certificate store counts. HTTPS is verified against your OS trust store as well as the bundled roots, the way pip does. Where IT has installed the inspecting CA in the OS store (on Linux: the system bundle that update-ca-certificates builds), hub calls and registration work.
• Nothing else changes: the bundled roots still apply, so a Linux box or container without a system CA bundle keeps working, and SSL_CERT_FILE is still honored.
• docs/troubleshooting.md says what to ask IT for when the CA isn't in the OS store.

Under the hood

• New dependency: truststore (pure Python), injected once at the CLI entry point. The hub and worker entry points are unchanged.
• Arena rebuilt because uv.lock gained that package, and classified at ARENA_MAJOR=2: the arena never loads it and nothing on the scoring path changed, so the verified corpus carries over.