Skip to content

Loki Mode v11.154.0

Choose a tag to compare

@github-actions github-actions released this 17 Nov 17:04
· 89 commits to main since this release
Immutable release. Only release title and notes can be modified.

A false VERIFIED, fixed

If you are on v11.145.0 to v11.153.0, upgrade. On those versions LOKI_E10_BASE_RED_BY_ID=0 gives the safe behaviour.

Measured with Loki's own safety tests (tests/min/run-safety.sh, no model): v11.153.0 ended 154 passed, 11 failed, six of them sealing VERIFIED on changes that must not be delivered; this release ends 165 passed, 0 failed, and putting the old default back fails 12 again. Replayed on three recorded FireLater runs: every already-failing check is still recognised (23 of 23, 23 of 23, 24 of 24), none counted as introduced. Not run on a model for this release.

  • Fixed a false VERIFIED. From v11.145.0 to v11.153.0 a run could be labelled VERIFIED when its change broke a test that was already failing on the base commit in a new way (an assertion failure on base, a TypeError after the change; an asserted URL that moved from GET /p to GET /admin/p): a failing test was excused as pre-existing by its id alone, on the claim that a test red before the change cannot be broken by it. The claim is false. An already-failing test is now pre-existing only when it fails with the same error as on base, or (full suite only) with an error that differs by noise alone (a path, a line number) and names no changed file or symbol. For the full suite and for an edited test file that is the rule from before v11.145.0. Two cases are new rather than restored: a test whose rerun on the base commit could not run is compared with the error its package's suite recorded on base (before v11.153.0 it always blocked), and a test beside a changed source file with the same name is excused on the same error (before v11.149.0 it was always held back). A changed number or quoted value, another kind of error, or no error line to compare blocks. LOKI_E10_BASE_RED_BY_ID=1 brings the id-only behaviour back and is unsafe for the label; =0 also stops reading a package's suite record when the base rerun gave nothing to compare. Measured: tests/min/run-safety.sh went from 154 passed, 11 failed on v11.153.0 to 165 passed, 0 failed, and with the id-only default put back 12 checks fail. The three recorded FireLater runs the id-only rule was added for still come out with every already-red check pre-existing and none introduced (23 of 23, 23 of 23, 24 of 24): rerun on copies of their trees, all 156 full-suite error lines and all 73, 73 and 97 targeted ones equal the base record. The run that led to the id-only rule (FireLater issue 24, sealed FAILED) was measured the same way: 150 of its 152 failing tests repeat their base error, and two Header test ids fail with a message that really changed after the change edited header.tsx (on base "Unable to find an element with the placeholder text", after the change a failed class assertion and "expected undefined to be 'BUTTON'"). That run is not excused again, and this is the intended outcome: a change that alters how an already-failing test fails is not shown to have left it alone.
  • A mock of a package is excused as "not a weakened test" only when the file that starts importing the package is the test's own subject: a file the test already imported by a relative path, or one with the test's name in its directory or the parent directory. Since v11.153.0 any source file in the change gaining the import excused a file-wide mock of that package in any test, including an existing test of another module.