Skip to content

Loki Mode v11.145.0

Choose a tag to compare

@github-actions github-actions released this 16 Nov 23:01
· 285 commits to main since this release
Immutable release. Only release title and notes can be modified.

The label is now checked by Loki itself

  • The label is now a count, not a reading. Loki gives the session numbered acceptance lines and two check functions (loki_http for a request, loki_cmd for a command); the proof script proves each line with them. When Loki re-runs the script itself, its own copy of those functions prints each result with a token that exists only in that run, and code counts them: a line is met when it has at least one passing check and no failing one. The same evidence always gives the same label, on any model. The model's reading of the change is kept as a note beside each line and as advice handed back to the session during the run; it no longer decides the label.
  • The same loop now covers the acceptance lines. Once Loki's own proof run passes, Loki checks the change against every acceptance line before the session is let go, and hands each unmet line back with the reason (for example a path the page calls that nothing exercised) so the session closes it and extends the proof. The final check at seal still decides the label.
  • Loki now closes the loop on its own proof run. After the session says it is done, Loki runs the proof script itself in a new empty directory; when that run fails, the session is resumed with the real failing output and asked to fix the product or the script, up to three times (LOKI_E10_PROOF_LOOP_MAX), before the run is sealed. A session on a cheaper model is corrected by evidence instead of being trusted.
  • The session that does the work on an issue now runs as Claude Code ships: its own system prompt and every tool it has (subagents, the task list, background shells), where Loki used to give it six tools under a replacement prompt with background work disabled. Loki adds the brief, the safety and the independent check around that session and no longer narrows it. LOKI_E10_FULL_SESSION=0 restores the six-tool profile. Plan, verify, fix and review sessions are unchanged.
  • A session that finishes without leaving the proof script is resumed once and asked for it, so Loki always has something to re-run. An unfinished session is now continued up to a 5.00 dollar line (LOKI_E10_CONTINUE_MAX_USD; it was 1.00, which a whole-issue run crosses before its first continue).
  • An issue written in prose (no checklist) gets the same label checks, from the acceptance lines the plan wrote; the receipt and the pull request table say which list was used.
  • A full-suite check whose every failing test was already failing on the base commit is now treated as pre-existing by test id, whatever its error text says (LOKI_E10_BASE_RED_BY_ID=0 restores the old rule). A finished run on a repository with a red frontend suite was sealed FAILED with no pull request because a stack trace of an already-red test named a file the change edited. A new failing test, a failing test that names a changed file, or one that also fails in an unexcused targeted check still blocks.
  • Loki now re-runs the proof itself. An issue run's session leaves one script ($TMPDIR/loki-proof.sh) that builds everything from empty and exercises each requested item; after the session and every fix round Loki runs that script by itself in a new empty directory and records the real exit code and output. The acceptance check reads that output, not the session's description, and a re-run that fails keeps the run from VERIFIED. LOKI_E10_PROOF_RERUN=0 opts out; LOKI_E10_PROOF_RERUN_S sets its time limit (default 600 seconds).
  • loki dashboard no longer crashes at startup: the Control Plane bundle could not find finding-schema.json (ENOENT) because the bundle sits in a different directory than the CLI bundle.
  • The green label on an issue run is now checked against the issue's own acceptance lines, word for word, by default (LOKI_E10_ISSUE_ACCEPTANCE=0 opts out). One short check marks each line met or not met: a line that asks for behavior is met only when it was exercised (a passing check, or a command and its result in the session's reply); code that was never run is "not proved by running". A repository that needs a database gets one more required line, that the change works on a fresh install. Any line not met makes the run PARTIAL and is named. The check runs on PARTIAL runs too, so every pull request shows each line's status.
  • The pull request now opens with Loki's label ("safe to merge" or "needs you") and a table of the issue's acceptance lines, each met or not with its evidence. The same label is set on the pull request and on the issue as a GitHub label (loki: safe to merge, loki: needs you; LOKI_PR_LABEL=0 opts out).
  • With an acceptance list, work the session left undone outside that list (an item the issue marks optional) is disclosed and no longer turns a finished run into PARTIAL. The session is told optional items may be skipped under LOKI_DONE.
  • The reviewer and the acceptance check judge the same list, and the reviewer is given the checks the harness already ran, so it no longer marks a line unmet only because it could not run a test itself.
  • The implement session of an issue run no longer inherits database or cache connection settings from your shell (DATABASE_URL, REDIS_URL, PG*, MYSQL_*, MONGO* and similar), so it cannot apply migrations to a database it did not start.
  • The session is asked to prove a fresh install (an empty database set up by the repository's own migrations, a first record created through the product's own code) and to list each requested item with the command it ran and what came back.
  • On an issue run with the Claude provider, an implement reply that ends with LOKI_SPEC_CONFLICT but cites no test file is treated as unfinished work: the session is resumed once instead of getting the conflict correction, and if it still stops the run is PARTIAL with its work and pull request kept, never BLOCKED (off by default until it is measured; LOKI_E10_CONFLICT_NEEDS_TEST=1 turns it on). On FireLater#24 (Haiku) such a reply had sealed SPEC_CONFLICT and left 13 changed files unpublished.
  • A resumed session that was auto-compacted is now priced: its cost record was rejected because the compaction summary counts in the cumulative usage but not in the call's own output, so the run cost read as unknown and the session-continue dollar line could not act (SDK route).
  • The prove stage stops what its prover session, its repair resumes and its script passes leave running by the same proven ownership as the implement session's scratch, never by its prove/pids file, and removes each scratch afterwards; an endpoint counts as the pass's own only when its listener is proven so. On FireLater run ps-1 the prover's own run had left a Postgres and a Redis running after the run.
  • The scratch.cleanup event is written on a normal run too (it was written only on a cancelled run, and the receipt entry only on a normal one); a cancelled run's exit-hook cleanup now reaches the event as well.
  • The scratch cleanup record (the scratch.cleanup event and scratch_cleanup in the receipt) names each process by the base name of its executable, pid, port and reason only, never its command line, which can carry a password or a token.
  • Scratch cleanup reads process start times in UTC on both sides (ps and the parser): with TZ exported, start times were shifted by the TZ offset and the cleanup silently found nothing to stop.
  • An issue run's implement session is told to use only a database, cache or server it started itself under $TMPDIR, never one named in the repository's env files or its environment, and never to apply migrations to one.
  • A VERIFIED run no longer asks for an optional rating on the terminal: it printed a prompt and read stdin for up to 30 seconds before exiting.
  • On an issue run with the Claude provider, the implement session is given a run-owned scratch directory (short path) as LOKI_SCRATCH and TMPDIR plus the run's own LOKI_RUN_OWNER value, and the harness stops what the session left running: every process of this user that started during the session and is proven the run's (the value in its environment, its working directory or a command-line path inside the scratch directory, or a parent that is the run's), never by a path in the work tree and never by the session's own pids file. Start times are rechecked before each signal, and the directory is removed only once nothing owned is alive. This happens after implement, in an exit hook, after a worker the supervisor stopped, and at the start of the next run for a record whose worker is gone. The receipt's scratch_cleanup lists what was stopped and any listed pid left running as not proven ours. The session is told to bind to 127.0.0.1 or a unix socket, to keep data directories, sockets and scratch files under $TMPDIR, and to leave what it started running for Loki to stop; the host guard's kill and rm allowances for the scratch directory are removed. A daemon that loses its environment and keeps nothing in the scratch directory (for example redis-server --daemonize yes with its --dir elsewhere) is left running and reported.
  • The host guard no longer refuses a file write such as cat > file <<'EOF' ... EOF (one or several in one command) because the body holds an apostrophe or a lone double quote ("command text could not be parsed safely"). A quoted heredoc's body is read as data only when cat or tee reads it, its output is not piped on, and no command line names a shell, eval, source, exec, xargs, ssh or coproc. Command lines and unquoted bodies are checked as before. On candidate 1's FireLater run this refusal blocked two multi-file writes.
  • On an issue run with the Claude provider, the implement session is no longer told to keep to a small, quick change: its rules let it start the repository's own services (stopping only what it started), its brief asks it to finish every item the issue lists, to prove the change by running it on a free port, and to end with LOKI_UNFINISHED: <what is left> when something is not delivered, and the time-budget sentence is left out. Database and cache servers the repository needs that are installed off PATH (for example a Homebrew keg-only Postgres) are added to the end of the session's PATH and named in the brief (LOKI_E10_ENV_CARD=0 turns that off). An unfinished run is PARTIAL with the pull request kept and what is left named. LOKI_E10_PROVE_BY_RUNNING=0 restores the previous issue-run brief; text goals and opencode runs are unchanged.
  • The prove stage's prover now runs its own setup and every ask script before it finishes and fixes defects in its own scripts (quoting, a request body the real caller does not send, data left over from its own run, scripts not safe to run twice) without changing what they check; the harness still runs the final scripts in a clean environment of its own. A plain prover session on a fully working tree passed 5 of 9 of its own first scripts and 9 of 9 after fixing them. A file the prover's own run creates in the tree is removed and disclosed; an edit to an existing file still caps.
  • An issue run with the Claude provider now gets a prove stage after implement and before verify (LOKI_E10_PROVE=0 opts out). A short listing call that sees only the issue fixes its asks, each a verbatim line of the issue (one retry; a second failure is PARTIAL "asks could not be listed"). A fresh session given the issue, the environment card, that fixed list and the working tree, never the implementer's reply, writes one script per ask that runs it against an empty environment built by the repository's own setup path. The harness runs setup, each script and teardown in a process group it owns with a per-script limit, checks that every endpoint setup opened is the run's own (not listening before setup, owned by a recorded process id or the run's temporary directory, started during the pass), stops recorded processes teardown left, and reruns the passing scripts on a checkout of the base commit. The run can be VERIFIED only when every ask's script exits 0 on head and fails on base; exit 3 is "not run here", a script that also passes on base is "check also passes without the change", and every other result is not proved, each capping the run at PARTIAL with the diff and the pull request kept. The stage has its own budget (LOKI_E10_PROVE_BUDGET_S, default 600 s) taken off implement and added to a sized run cap. The receipt carries a prove record (with wall seconds), and the pull request body opens with a table of ask, command and result. On FireLater#24, 13 of 22 attempts scored 6 of 9 or less when a blind scorer started the product, with tests, lint and typecheck green on 21 of 22; 4 of the 7 labelled VERIFIED were broken. A text goal, an opencode run and the opt-out are unchanged.
  • The prove stage now sends asks its scripts did not prove back to the implement session by same-session resume, with each failing command and its output, at most two rounds inside the implement time left, and reruns the same scripts after each (LOKI_E10_PROVE_REPAIR=0 opts out). An ask skipped by the endpoint check is not sent.
  • On an issue run with the Claude provider, the implement session is asked to prove its change by running it: start the service, call each new or changed entry point the way its real caller does, create a record through the repository's own code where it has one, then stop only the processes it started (LOKI_E10_PROVE_BY_RUNNING=0 opts out). On FireLater#24, Sonnet trees with green tests sent new client calls to paths that 404; two plain Sonnet runs given this instruction scored 8 and 8 of 9 blind against 3 and 6 without it, at about 31 percent more cost. Text goals and opencode runs are unchanged.
  • A run whose reviewer says an asked-for line is unmet (UNMET: ...) is never sealed VERIFIED: it is PARTIAL with that line in NOT PROVEN, and an UNMET line is no longer cut when the reviewer writes more than five reasons. On FireLater#24 (Haiku, n=1) such a run had sealed VERIFIED.
  • A rerun of loki start owner/repo#N stops before any model call when an open Loki pull request from the same owner already covers the issue, and names it, instead of opening a second one (LOKI_RERUN_GUARD=0 starts the run anyway). The existing pull request is never touched.
  • An implement session whose final text ends without a finish line (for example a compacted session that writes LOKI_BLOCKED: ... still outstanding) is no longer sealed as done. On the claude provider it is resumed in the same session, up to 4 times while each resume changes the tree and the run has spent under $1.00 (LOKI_E10_CONTINUE_MAX, LOKI_E10_CONTINUE_MAX_USD). If it still has no finish line, implement exits incomplete: the work and the pull request are kept, the run is PARTIAL with the outstanding work named, and it is never VERIFIED. On FireLater#24 (Haiku, n=1) such a run had sealed VERIFIED.
  • loki start owner/repo#N runs a large issue as one whole-issue run instead of splitting it into slices up front, on Sonnet and stronger models (LOKI_ISSUE_WHOLE_FIRST; 0 restores the split, 1 forces the whole-issue run on Haiku too). A run that implements on Haiku keeps today's split. loki backlog and loki issues run children follow it, and backlog triage now says a large issue runs as one run. On FireLater#24 (Sonnet, n=1) the split route did 5 of 10 asks for $3.45 and the whole-issue route 8 of 10 for $0.66.
  • New, off by default until measured: LOKI_E10_VERIFIED_FINISH=1 runs the pre-delivery review as a fresh finish session that sees only the diff, the changed files' pre-fix text and the repository's contribution rules, instead of resuming the whole implement session, and always reruns the touched tests afterwards, taking the finish back if it broke one. The receipt says when it fired.
  • New, off by default until measured: LOKI_E10_REPO_CONVENTIONS=1 checks the change against the repository's own conventions before delivery. Its pre-commit hooks that need no install (file-name rules and regex greps) run on the changed files, and the contribution guide's explicit "never use" or "do not use" rules are checked against the added lines. A violation goes to the pre-delivery review with the hook's message or the rule.
  • New, off by default until measured: the pre-delivery review turn can be kept from breaking a test. With LOKI_E10_REVIEW_TURN_GUARD=1, when the turn edits anything, Loki reruns the tests of what it changed; if a test that passed before the turn fails after it, only that turn's edits are taken back, the implemented change is kept, and the receipt says which test it broke.

Measured

  • One issue in one repository (FireLater issue 24, nine acceptance lines plus the fresh-install line Loki adds), run exactly as a user types it, Sonnet, seven runs in a row while this was built. The last run, on this code: VERIFIED, 10 of 10 lines met by count, 34 passing checks and 0 failing in Loki's own run of the proof (16 seconds, a fresh database, the first tenant registered through the sign-up route), 2.04 dollars, 11.9 minutes, nothing left running. The three runs before it, judged by a model's reading instead of a count, read 9, 8 and 7 of 10 on the same kind of evidence, which is why the count replaced the reading.
  • The released 11.143.0 labelled a run of this issue VERIFIED that broke creating new tenants. The acceptance check rejects that same change: six lines not proved by running.
  • Not measured: any other repository, any model other than Sonnet, opencode.

Known gaps

  • The session writes the checks. Loki executes and counts them, and a reviewer reads them for gaps, but a thin check can still pass a line. The reviewer's notes are on the pull request.
  • An issue with no checklist uses acceptance lines Loki's plan wrote; the pull request says so. If no plan ran, the run gets the old label.
  • A page is proved by its requests and its tests, not in a browser.
  • A run costs about 1.7 to 2.5 dollars on this issue, more than before the label work.
  • Tests already failing on the base commit are ignored by test id; a change that breaks an already-broken test further is not detected.
  • A failed run with committed work did not open a draft pull request in one live run; not yet traced.
  • The repository's own test suite has not been run on this release: several tests pin text and defaults this release changes and are expected to fail until updated. The founder ordered releases without test waits.