Skip to content

v1.9.0 — No empty PASS, define mistakes named

Choose a tag to compare

@yottayoshida yottayoshida released this 07 Oct 02:57
· 302 commits to main since this release
97bc80a

Added

  • sideeye completions zsh|bash|fish prints a completion script built from the usage lines (#712; src/cli.zig, docs/cli.md). It offers exactly the commands, a command's positional argument (a case file, a command name, a shell), and exactly the flags its usage lines list (explore's and preflight's two lines together, so --config and --state are both offered), with a flag's value completed as a file where the usage names a path and as its listed words where it lists them (--observe). The bash script runs on bash 3.2, macOS's /bin/bash, quotes a file name with a space, gives a directory its slash (except one named after a :, = or @ in the word, which completes without it), and completes a path holding :, = or @. spike/check-completions.py reads the usage lines on its own and asks each shell what the script offers at every position; acceptance runs it on Linux with all three shells installed, the macOS job with /bin/bash and zsh. A Homebrew formula can take the scripts with generate_completions_from_executable; the tap is not changed here.

  • On a terminal the verdict word is coloured — PASS green, FAIL red, UNKNOWN yellow, SETUP ERROR magenta — and nowhere else (#712; src/report.zig, docs/cli.md). Only standard output is asked, so a report redirected into a file from a terminal carries no escape sequence of Sideeye's own; NO_COLOR set to anything non-empty, or TERM=dumb, turns it off on a terminal too. The JSON report, the MCP server and every captured run (CI, acceptance) are unchanged. A CI that gives its steps a terminal and greps the text report should set NO_COLOR=1.

  • The README's first screen shows a trimmed real sideeye demo FAIL report, and CI holds every line of it to a fresh demo (#714; spike/check-readme-demo.py). Eleven lines under the opening paragraph, each one the demo prints, in its order, with the scratch directory shown as …; the macOS job's demo step and acceptance check 5 run the demo and compare, so a change to the wording of a line it shows turns CI red until the README follows. The one named exception to "never a silent PASS" — a directory a recorded rename moved in from outside the judged tree — went from two sentences to a clause of the promise's own sentence, with its explanation moved to DESIGN.md's known constraints, and spike/check-readme-shape.sh now holds the exception to that sentence (ADR 0032).

  • docs/found.md lists every report in spike/upstream-reports.tsv — the reports this project has filed on GitHub — with the tool, a link and the furthest state the ledgers record; generated, and compared in CI (#713; spike/outcome-funnel.py, ADR 0070). 35 reports today: 5 fixed upstream and re-measured against the fix, 1 fixed upstream, 28 filed, 1 withdrawn. The page is rendered from spike/upstream-reports.tsv and spike/outcome-funnel.tsv by the same --write-doc that renders the funnel's counts, and --check-doc now compares both pages. The README stops quoting a count of reports (it said 27) and links the page; spike/upstream-report-template.md does the same. check also refuses a report the two ledgers disagree on whether it was withdrawn.

  • The four targets #684 named, re-measured past the calls that change nothing: fish PASS, dotdrop and firewalld FAIL with no content lost, vim at its next wall (spike/dogfood/2026-10-06-read-only-684/; docs/target-classes.md, spike/dogfood/RUNS.md, spike/outcome-funnel.tsv). A strace-only survey written before the engine changed predicted each answer: vim writes the file's ACL back after saving and now stops on setxattr, in all three observation modes; fish's universal variables PASS 6/6; dotdrop install -f FAILs 2/15 and firewall-offline-cmd FAILs 1/6, each truncating a file before writing it, with the old bytes kept in its own backup (.dotdropbak, .old). Neither FAIL is reported upstream.

  • Thirty tools whose state git does not hold, in six rounds of five on the released v1.8.0: 22 FAIL, 8 PASS, and four reports upstream (spike/dogfood/2026-10-05-user-data-2/; docs/target-classes.md, spike/dogfood/RUNS.md, spike/outcome-funnel.tsv, spike/upstream-reports.tsv, spike/unknown-rate/b2-exclusions.txt). 359 names were screened, none at a project already reported to, and 57 went through the entry gate. Twenty-one FAILs are the truncating open and the kill before the write. SolveSpace's solvespace-cli regenerate empties the sketch and still reports success, PyMOL's save empties the session, WP-CLI's config set empties wp-config.php, and ffsubsync's --overwrite-input empties the subtitle; they are reported as solvespace/solvespace#1783, schrodinger/pymol-open-source#520, wp-cli/config-command#233 and smacke/ffsubsync#240. Fourteen of the thirty are static images named bare, and every one reached a verdict through v1.8.0's next step to --observe supervised. aliyun-cli, a static image whose child is dynamic, did not: the page's path ends at oracle_missed_operation and then at syscalls with no mode named, the shape lefthook met on v1.7.0. Two screens missed: fresh.sh did not read cohort 4's rejected candidates (ffsubsync), and uv was screened under a spelling no ledger carries; both left the slate with their explores kept, and go env -w took uv's place. The B2 exclusions now name each tool by the trixie package that ships it, looked up rather than recalled.

  • sideeye preflight --config sideeye.toml reads the toml as explore reads it, and refuses the define mistakes it can see before any crash world, in explore's words (#704, ADR 0094). Preflight refused --config ("sideeye explore --config answers strictly more"), so a define moved into a file — the README's next step after preflight — could no longer be asked the quick question, and a check that could not start or a marker the operation never printed surfaced only from an exploration. The refusal is gone; the toml goes through the same phases explore's does, so the detectors already there apply as they stand: the parser, the cwd, apparatus and scratch checks, a state that is not a directory, a command that cannot be started, a marker the recording never printed, no crash point, and the cwd a toml's define needed. Two of explore's refusals are added to preflight, at the point explore reaches them: with a declared check and a crash point, a check with no command in it is refused --check is empty, and one against an initial state with nothing to corrupt checker_not_falsified — each one function, refuse.checkerArgv and refuse.refuseNothingToCorrupt, now called from both. docs/cli.md lists what is refused, one mistake per run as in explore; what only a crash world shows is not on it. Nothing the toml declares is run: the report adds marker, checker and recovery lines saying what preflight did with each, and a recovery that cannot be started is not refused here, as it moves no verdict in explore (ADR 0072). The next line is sideeye explore --config '<the toml>', quoted for /bin/sh and named in the directory its relative paths were resolved against, so a toml that is a symlink keeps its own side; a toml read from /dev/stdin, from a shell's <(…), or through a link into /dev or /proc gets a line saying to save it first, and one whose name cannot be printed as typed a line saying to copy it to a file with a plain name. --twice works on a toml and names [world] state. With the flags, preflight accepts and refuses what it did and its accepted report is byte for byte what it was; it still refuses --check, --marker and --recovery by name, reworded to point at --config, and define flags beside --config get explore's "mutually exclusive" refusal. The synopsis has a preflight --config line, and #273's check covers it. spike/acceptance.sh check 6c runs one toml per listed mistake through both commands and compares the exit code, the refusal and its detail, with the check counting its own starts; against the parent commit with only the --config refusal removed, twelve of its legs fail, and #273's check with them.

Changed

  • A string-form command spelled for a shell runs as written, and is told so (#706, ADR 0095; src/config.zig, src/report.zig, src/main.zig, src/refuse.zig, src/mcp.zig, docs/cli.md, docs/report-schema.md, docs/contract-freeze.md). A string-form command is split on spaces and never run by a shell, so operation = "./kva put 'hello world'" handed the program two arguments with the quotes in them, "… && echo ok" handed it && — and both PASSed with nothing said. Now a setup, operation, check or recovery command whose string holds a quote, or a whole word a shell would act on (&&, |, ;, a redirection, a word beginning $ or ~/), earns one sentence naming the command and how Sideeye reads it, with the argv form that groups what quotes meant — none where a shell would have expanded a $ or backquote inside them. A toml command value cut at an inner quote by a # (check = "grep -c "#include" f" runs grep -c ) is accepted as before, and the sentence says what it became. It is in the report (warning), the JSON (define_warnings, a new optional field), sideeye mcp's summary, and on stderr as sideeye: warning: …, written once at the end so the first line of merged output is still the verdict — and so a run that ends in a SETUP ERROR, or a preflight, shows it too. Nothing runs differently and no spelling is refused. A symbol inside a word (x=n*10, <esc>, ~>1.6) is left alone, and a replay is not looked at. A toml value refused as "not one double-quoted string" (expected_status = 0, state = './state', operation = "tool "two words"") is refused with the line it would have been read as.

  • Every re-emitted checker line says which world it came from (#707; src/main.zig, src/recovery.zig, docs/cli.md, docs/evidence.md, ADR 0071 amended). A world's checker output came out bare, so a checker that printed a traceback put fifty lines ahead of the verdict that no reader could attribute to a world. Each line now carries world N: — N the crash point, the number of the report's crash point N of n — or baseline: for the un-killed world, and a recovery's lines recovery world N: and recovery world N check: , where two exhibits' legs had shared one mark. The checker's stderr is in the same capture, in order, and comes out on Sideeye's standard output with the same mark; docs/cli.md now says so. The terminal text changes; <work>/checker-output.txt, the evidence bundle and the JSON report do not. When the capture cannot be opened the checker runs uncaptured and its output is unmarked, as before.

  • An exploration in which no world could have failed is UNKNOWN nothing_could_fail, not PASS (#682, #683, ADR 0091; src/contract.zig, src/engine/snapshot.zig, src/refuse.zig, src/report.zig, src/main.zig). The README's rule for a checker — one that cannot fail makes the run UNKNOWN — now holds for the built-in layers too. A run with no crash point, whatever was declared, and a run with no checker, no crash world that printed the marker over a path only one snapshot holds, and no judged path a crash point could change, used to PASS with exit 0; they exit 2 now. "Could change" is a union: the path ended other than it began, or a crash point other than an fsync or a mkdir named it, or a rename named a directory above it — so an atomic rewrite with the same bytes still PASSes, and a parent's fsync or an os.makedirs(…, exist_ok=True) does not make an untouched path count. preflight refuses a recording with no crash point with the same reason — the one explore reaches once its completeness gate is met, since preflight has no such gate; replay is not asked, since its one world answers whether that world still fails. Three new next steps say where an undeclared define's commands run (nothing_in_state), or to declare a check — or a marker, where one has a created or removed path to judge. A record on a link itself (unlink, rename, link, symlink, rmdir, mkdir) names the link and not its target, both here and in the per-path reconciliation, where the record that made a link used to account for an unrecorded change at the file it points to. The unknown_reason closed set goes from 34 to 35 — the fifth break of docs/contract-freeze.md surface 2, recorded there by owner ruling; contract_version does not move. A read-only control row, such as topydo's ls used as a zero-operation PASS, now answers nothing_could_fail; records taken before this are left as measured. It is a floor, not a guarantee: a crash point that names a judged path without changing it — a lock file opened for writing, a failed call — still counts.

  • A PASS whose atomicity invariant judged only paths the operation never touched says so on its verdict line, and the JSON counts them (#683, ADR 0091). An exploration's such PASS was carried by a checker or the marker; the headline keeps its claim and adds ", but the operation touched none of the N path(s) it judged" (or ", but it had no path to judge"). The new optional field l0_judged_paths_touched comes after the two l0_ fields it qualifies. A replay's PASS carries both.

  • A state that exists and is not a directory is SETUP ERROR define_invalid, before anything runs (#682). It used to snapshot as an empty tree, and the run PASSed with no crash point.

  • Acceptance legs that measured observation through a PASS read nothing_could_fail instead, and the macOS kill-landing and fs_usage toys rewrite a file the state already holds (spike/acceptance.sh, .github/workflows/ci.yml, spike/thread-kill-lands.sh, spike/fsusage/acceptance-local.sh). The refusal is the last gate before a PASS and is raised after every world, so reaching it is the same evidence the PASS was; the counts those legs pin are in the JSON on both. Check 2nf is new and holds the rule's three falsifiable shapes.

  • The strace oracle reads extended-attribute reads, inotify_add_watch, preadv/preadv2 and the polls on the state directory as reads (#684; src/oracle.zig). getxattr, lgetxattr, fgetxattr, listxattr, llistxattr, flistxattr, inotify_add_watch, preadv, preadv2, poll, ppoll, select and pselect6 join faccessat2 and epoll_ctl (#542) on the oracle's read-only list, so a target that asks one of them about its state is no longer refused unsupported_syscall_observed for asking. These are the calls docs/report-schema.md named as "still refused"; that sentence is replaced. The writing side — setxattr, removexattr and their l/f forms — still refuses, held by a unit test and an acceptance control (TOY_SETXATTR). It applies in every observation mode, since the oracle is strace in all three; the macOS fs_usage oracle keeps its own list.

  • A mistake on the command line is named in one line, and each command has its own help (#705). The first answer to a mistyped command or flag used to be the whole reference — 218 lines, the longest 412 columns — with no line saying what was wrong: sideeye explor, sideeye init, sideeye --version and a bare sideeye replay printed it with exit 3, explore --frobnicate typed last was told "an option is missing its value", typed with a value it was "unknown option" without saying which, and demo, mcp, version and evidence refused an extra argument without naming it (evidence read --frobnicate as a file). Now each of those is one line that names what was typed and, when one is close, the nearest spelling that command accepts — explore --stat x is offered --state, explore --recover x --recovery, explore --twic x nothing, because --twice is preflight's. A value is named the same way: --observe bogus, --expect-status abc, --world-timeout abc, --apparatus and --scratch refusals now carry what was given, and --observe syscall is offered syscalls. A word that names no command, and an argument help, version, mcp or evidence does not take, goes to stderr with exit 3, as those argument refusals always did; a mistake among the flags of explore, preflight, replay or demo is still a SETUP ERROR line on stdout. A token the terminal would act on is defanged before it is shown. evidence still reads a path that begins with a dash when the file is there, and refuses it as the option it looks like only when it is not. sideeye help <command> and sideeye <command> --help print that command's own help — its synopsis lines broken to fit, its paragraph, and one line per flag it takes: the first sentence of that flag's entry, and every sentence of the entry that begins Caution: — cut out of the same text sideeye help prints, so the two cannot disagree. Three entries of that text were reworded for it: --twice rewriting --state and --observe syscalls killing a static helper now begin Caution:, and --oracle-fs-usage says it needs root in its first sentence, so the command's own help does not hide what <command> --help used to show in full. sideeye help still prints the whole reference, otherwise unchanged but for the synopsis line sideeye help [<command>]; sideeye --version is sideeye version; a bare sideeye prints the version line and the commands instead of the reference, still with exit 3. --version, help <command> and <command> --help for mcp, help and version exit 0 where they exited 3, which docs/contract-freeze.md §3 allows: exit 0 is the success of what was asked, and none of them produces a verdict. Help typed anywhere but alone right after the command (explore --state X --help, explore --help --state X) is refused with where help is answered, not answered: the parser may already have removed the --json report by then (#296). spike/acceptance.sh check 15 now holds <mode> --help to the mode's own help — shorter than the reference, starting with the version line, a summary line for every flag on the mode's synopsis line — and reads a table of mistakes for the words they must name, and #273's loop over mcp, help and version reads each refusal for the flag it got.

  • The CI quickstart pins v1.8.0 (docs/ci-quickstart.md, docs/ci-quickstart/release/install-sideeye.sh, .github/workflows/quickstart-release.yml). The release lane installs the published v1.8.0 asset instead of v1.7.0's. The pin moves after the release, as on v1.7.0, because the lane downloads the asset and the asset exists only once the release is published. Nothing else in the lane changes.

Fixed

  • The setup receives SIDEEYE_STATE_DIR, and the docs name it as where every define command finds the state directory (#708; src/main.zig, README.md, docs/cli.md, docs/checker-cookbook.md, docs/ci-quickstart.md). The operation, the check and the recovery already had it; the setup had only TOY_STATE, the demo toy's name, and the only variable the docs described was that one — a checker written from the README alone located the state relative to its own path. TOY_STATE still reaches every command with the same path, so nothing written against it changes. A SIDEEYE_STATE_DIR in the caller's own environment no longer reaches the setup: the engine's value replaces it there, as it already did for the operation and the check.
  • A setup, operation or check that cannot be started is a SETUP ERROR naming the file, raised before it runs (#701, ADR 0092; src/image.zig, src/main.zig, docs/cli.md, docs/report-schema.md). The engine used to spawn the command and read the failed exec's 127 as a fact about the target: an absent or mode-644 checker passed the falsification gate and then refused baseline_violates_invariant — a judgement by a checker that never ran — an absent operation refused recording_run_failed pointing at --expect-status, and an absent setup "exited 127" without naming a file. Now the run stops as environment, saying which command, which resolved file, and whether it is missing, not a regular file, not executable by this user or could not be examined — or that the #! interpreter it names is, or that a bare name has nothing on PATH that would start (execvp moves past a file whose interpreter is missing, so this does too). The setup is asked about before it runs, leaving nothing on disk when refused; the operation and the check after the setup, which may build or write them — a checker the operation itself would create is now refused, so write it in setup. Exit codes move for these inputs: an absent or mode-644 checker 2 → 3 (0 → 3 where the run recorded no operation and so never ran the checker), an absent operation 2 → 3 under explore, replay and preflight; an absent setup stays 3. The [recovery] commands are not asked about: a recovery changes no verdict and no exit code (ADR 0072).
  • On macOS, the next step Sideeye prints is one that runs on macOS (#702; src/main.zig, src/refuse.zig, src/cli.zig, README.md, docs/cli.md, docs/mcp.md). --oracle runs strace, which macOS has none of; it used to be accepted there and fail as "not an executable file" — or, given a real executable, run it as the witness — and the README's own explore line, preflight's next hint and the no-oracle refusal all named it. Now --oracle on macOS is refused as platform_unsupported before anything runs, naming --oracle-fs-usage and --allow-unverified under explore and replay and saying to drop it under preflight (which takes neither); preflight's note and hint end with --allow-unverified, and the hint, with its --check placeholder taken out, reaches PASS; completeness_not_verified names --oracle-fs-usage where it named --oracle; boundary_without_oracle says on macOS that no witness this engine reads can account for another process, where it said to pass --oracle (Linux); the README carries a macOS explore line beside the Linux one, labelled in the prose rather than with a trailing # macOS, which zsh would hand to sideeye as an argument. Exit codes and reasons move on macOS: --oracle naming a real executable 2 → 3 (it used to run that file as the witness and refuse later), --oracle naming no executable file stays 3 with setup_error_reason environment → platform_unsupported. On Linux every run's output is unchanged; --help now says --oracle is Linux only, and the --oracle-fs-usage refusal there, like this one, undoes the state directory the run had just made.
  • A toml with no cwd, run from elsewhere, is told which line to add (#700, ADR 0093; src/refuse.zig, src/main.zig, src/contract.zig, README.md, docs/cli.md, docs/report-schema.md). A toml resolves its own paths and argv[0] against its directory and every other argument against the directory the commands run in — Sideeye's own when no cwd is declared — so a define in the README's shape with a relative argument failed from any other directory, as --setup exited 1 or recording_run_failed pointing at --expect-status. Now, when the command that failed carries an argument present under the toml's directory and absent where it ran — or, for a file it is about to create, a directory above it that is — the refusal says so and names cwd = ".": a new next_step, declare_cwd, on recording_run_failed, marker_never_observed, nothing_could_fail with no crash point, checker_not_falsified (the corrupted state accepted) and baseline_violates_invariant's checker layer, and the same in the sentence of setup_failed and, under --observe syscalls, beside the mode's own step. An observation, not a guess at the cause — ADR 0086's reason for leaving the wording alone stands, and ADR 0093 records why this is not the guess it declined. The README's toml example declares cwd = ".", so a define copied from it means the same thing from any directory. Not seen: a relative argument that finds a different file and succeeds, one that names nothing on either side, a config read through a link or a pipe.
  • On macOS, refusing a framework Python's launcher names the interpreter to run instead (#703). python3 tool.py with Homebrew's Python, or with the Command Line Tools' framework bin/python3 — and a console script such as black, whose #! line names that Python — was refused child_process_detected with the step "check whether the operation is a shell script wrapping another command", which sent the reader looking for a wrapper that was not there. bin/python3 in a framework build is CPython's pythonw.c launcher: it replaces itself with Resources/Python.app/Contents/MacOS/Python through posix_spawn with POSIX_SPAWN_SETEXEC, which the shim's posix_spawn wrapper records only when it returns, so the chain of observation breaks. The refusal stays. On macOS its message now adds what was read — the image, read before the run; and at the refusal the script and the options on its #! line when that line names the launcher, the framework's interpreter and, when a pyvenv.cfg stands in the launcher's directory or the one above, the __PYVENV_LAUNCHER__ value the launcher would have set — and the new step name_framework_interpreter says to make that interpreter the operation's first word, followed by those options and the script, and to set the variable in the environment Sideeye runs in, since an env: apparatus entry is checked, not applied. The reason, the verdict, the exit code and contract_version are unchanged, and every other broken chain keeps the wrapper step — a --copies virtual environment's python among them, which is a copy outside the framework's bin. Two other ways in are not this refusal and keep their steps: /usr/bin/python3 on a Mac with only the Command Line Tools, and a script whose #! line is /usr/bin/env python3, are Apple-shipped binaries the shim never reaches (no_shim_marker). The layout is the one the launcher relies on, read rather than listed: a Mach-O in a bin directory beside the framework's library, which is named after the framework (Python, the Command Line Tools' Python3, python.org's free-threaded PythonT), and an interpreter named after the framework or Python. Measured on macOS arm64 with Homebrew's python@3.14 and @3.12 and the Command Line Tools' 3.9: each of python3 script, a virtual environment's python script and a console script was refused with the interpreter named, and the define rewritten as the refusal said was accepted, and reached PASS under --allow-unverified for Homebrew's, the virtual environments' runs writing their own sys.prefix. python.org's 3.13.7 installer was inspected and not installed: its launcher and interpreter sit in the same layout, and their signatures carry the entitlements that let the shim load. A new macOS CI leg, spike/check-framework-launcher-macos.sh with a stand-in launcher spike/toys/pythonw-launcher.c, holds the shipped engine to it: the launcher refused with the interpreter named, the interpreter judged, and the wrapper step kept once the framework library is gone. The README's target limits and DESIGN.md's known constraints say it, and spike/check-readme-shape.sh's bound moves by the line's 414 bytes.