Skip to content

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 25 Oct 10:24
Immutable release. Only release title and notes can be modified.
0873ce2

Added

  • The accept label, the accept-label Action input (default pharn/accepted, at most 50 characters). In
    test-integrity: block on a pull_request or pr-number run, a pull request that carries the label, applied last by
    a user account that is not the PR author, has its block signals downgraded to warn: the report gains the
    TEST_INTEGRITY_ACCEPTED message (info), testIntegrity.accepted and, for a non-default name,
    testIntegrity.acceptLabel, and the comment says who accepted it. A label that does not count (the author or a bot
    applied it, no labeling event, or anything unreadable) adds TEST_INTEGRITY_ACCEPT_IGNORED (warn) and the blocks
    stand. The Action reads the pull request and its issue events (at most 10 pages of 100) with pull-requests: read or
    issues: read; the CLI never honours the label, and test_integrity.accept_label in .pharn/ci.yml is refused. The
    TI03, TI10 and TI13 hints now name the label.
  • The test-integrity Action input and --test-integrity CLI flag (block, warn or off, default warn),
    and a test_integrity.mode key in .pharn/ci.yml that can only make the check stricter than the input. The
    report's testIntegrity.mode and testIntegrity.source say which mode applies and where it came from, a
    looser config value is ignored with the new TEST_INTEGRITY_MODE_IGNORED message (warn), mode off adds
    TEST_INTEGRITY_OFF (info), and the PR comment and job summary gain a "🔍 Test integrity" section.
  • Signal TI01, test file deleted: a deleted test file whose removed lines define tests, or a test file renamed to
    a path that is not a test file (edited or not, since a rename shows only the lines it edits), is reported in
    testIntegrity.signals and in one TEST_INTEGRITY_TI01 message,
    block in mode block (a failing check), warn in the default mode. It is info when the same change also
    deletes the source file of the same name, deletes other files of the test's package, or adds a test file of the
    same name (a migration); helpers, fixtures and an __init__.py under tests/ do not count. The "🔍 Test
    integrity" section lists the findings. Test files are now recognized by the spec's globs: .test. and .spec.
    with a JavaScript or TypeScript extension, .e2e.ts, .pw.ts, test_*.py, *_test.py, *_test.go and the
    test, tests and __tests__ directories. TEST_FILES_DELETED is unchanged and now uses the same detection.
  • Signal TI14, diff-coverage exclusions widened: a change to the pharn-ci config in use (.pharn/ci.yml, or the
    config input) that lowers or removes coverage.threshold, or adds a coverage exclusion, is reported in
    testIntegrity.signals and in one TEST_INTEGRITY_TI14 message, block in mode block, warn in the default
    mode. Raising a threshold, removing an exclusion and unrelated settings never fire. CONFIG_CHANGED_IN_DIFF is
    unchanged.
    The config lines are read as the config reads them: a quoted key, a quoted or escaped value and any number of
    decimals all count.
  • Signal TI11, CI stopped running tests, from the diff of workflow files (.github/workflows/*.yml|yaml),
    reported in testIntegrity.signals and in one TEST_INTEGRITY_TI11 message. block in mode block: a removed
    test command (pnpm test, vitest, jest, pytest, go test, ...) that no workflow adds again, and a removed
    pull_request trigger. warn in every mode but off: a shrunk flow-list matrix, an added
    continue-on-error: true, an added if: that is a literal false or leaves pull requests out, and a
    timeout-minutes of 0 or 1. Which step a line belongs to cannot be seen without context lines, hence the warnings.
  • Signal TI03, a skip added to a test file, from the kept text of the diff, reported in testIntegrity.signals
    and in one TEST_INTEGRITY_TI03 message. It reads JS/TS (.skip, xit and the other x heads, .todo, .fails,
    skipIf, runIf, this.skip(), Python (@pytest.mark.skip, skipif, xfail, @unittest.skip*, pytest.skip() and
    Go (t.Skip(, t.Skipf(, t.SkipNow(). An unconditional skip of a test the diff shows existed is block in mode
    block; a conditional skip, a skip inside a test body, and a skip in a test the change adds are warn. A skip that
    was already there (an edit, a move) never fires, and environment guards such as skipIf(!hasCamera) or
    pytest.skip("REDIS_URL not set") stay warn.
    At most 20 findings per file, with the skips of existing tests kept first, so warn findings never push a block out.
  • Signal TI04, a focused-run marker added to a JS/TS test file (.only, bare fit( or fdescribe(), reported
    in testIntegrity.signals and in one TEST_INTEGRITY_TI04 message, block in mode block, warn in the default
    mode. solver.fit(x), def fit(self), comments, strings and files that are not tests never fire; a marker that was
    already there (an edit, a move) does not fire either.
  • Signal TI10, a test script weakened, from the diff of a modified or renamed package.json (root or nested),
    reported in testIntegrity.signals and in one TEST_INTEGRITY_TI10 message, block in mode block, warn in the
    default mode. In a script whose key starts with test, an added || true, an echo or exit 0 as a command,
    --passWithNoTests, --testPathPattern and -t or --testNamePattern each fire once per line, unless a removed
    line of the same key already had that form; a removed pretest or posttest script fires too. --passWithNoTests
    stays quiet when no tracked file is a test file. Other scripts, an added or deleted package.json and a pure edit
    that keeps the weakening never fire.
  • Signal TI13, a coverage-ignore comment added to source, reported in testIntegrity.signals and in one
    TEST_INTEGRITY_TI13 message, block in mode block, warn in the default mode. An added Istanbul, V8 or C8
    ignore hint in a comment of a JS/TS file, or a # pragma: no cover in a Python file, that is not a test file fires;
    a marker that was already there (an edit, a move) never does.
  • The unified-diff parser keeps the text of added and removed lines per hunk for test files, the pharn-ci config in
    use, workflow files, every package.json and the JavaScript, TypeScript and Python source files (at most 5000 lines
    per file, 400 characters per line; textTruncated marks a cut file), for the signals that read line text. readDiffChunks takes an optional { configPath, repoRoot } so a custom
    config path works through the stream as it does through check().
  • The accept-label input also refuses invisible formatting characters (zero-width marks, bidirectional overrides
    and isolates, a byte order mark inside the name), which could make the hint around the label read differently from
    what it says. A spelling of pharn/accepted in other letter case is read as the default, so the report, the hints
    and the sign-off line name the same label. The comment shows those characters in a path or label as U+FFFD.
  • The PR comment and job summary follow the Report v2 layout. The summary table gains a Test integrity column (off,
    none (warn), or the number of findings and how many block), and the headline sentence counts the findings when
    there are any. The "🔍 Test integrity" section lists block findings first, then warn, then info, shows 10 and
    collapses the next 10 in a <details> block. The Messages section leaves out each TEST_INTEGRITY_TI* message
    whose findings the section lists, so a finding and its hint appear once; report.json keeps every message.
  • Precision of the diff-only signals, from a review of the base-rate fixtures. TI11 reads a quoted "on": key and the
    test commands npm t, pnpm -r test, pnpm --filter <pkg> test, pnpm vitest, pnpm exec, bun x,
    node --test, bun test, deno test, tox, nox and the uv, poetry, pipenv, pdm, hatch and rye run
    wrappers. TI10 reads ; true, || : and ; : as ignoring the exit status, no longer reads a banner
    (echo start && vitest run) as a weakening, and no longer reads mocha's -t (its timeout) as a name filter. TI13
    reads coverage.py's # pragma: nocover. TI03 reads a Python skip decorator above an edited def line, or above an
    existing test in a diff with context lines, as a skip of an existing test. TI01 no longer counts a deleted fixture,
    snapshot or data file under a test directory because its text was cut. A hand-made hunk header that starts a side at
    line 0 numbers its lines from 1.
  • The TEST_INTEGRITY_TEXT_TRUNCATED message (warn, not in mode off): a file that a test-integrity signal reads
    (a test file, a workflow, a modified package.json, a JavaScript, TypeScript or Python source file, or the pharn-ci
    config) has more than 5000 changed lines, so the signals read only the first 5000. The message names those files,
    so a change cannot pass in silence by padding a file past the cut. Deleted files are not named.
  • README: an Agent skills section on the pharn-ci plugin in
    pharn-dev/skills (install, explain a verdict, cover
    uncovered lines), and a pointer to it from the Quick start and the Pharn family table.
  • The Skylos nine benchmark: the nine ways Skylos hides a failing test, reproduced in vitest under
    fixtures/skylos-nine/ and pinned by tests. The diff-only signals of this release (TI01 and TI03) catch 2 of 9;
    the other seven are named with the planned signal that would catch them.
  • A "What test integrity does not catch" section in docs/LIMITS.md, and a "Test integrity"
    section in the README with a rendered example of a finding. The "Only added lines count" limit no longer says that
    skipped tests are not detected.
  • A release smoke script, scripts/smoke-release.mjs, that builds a throwaway repository from fixtures/smoke/, runs
    the built Action bundle against it with no token and no event, and checks the exit code, the verdict output, the
    job summary and the footer version. docs/RELEASING.md describes running it in the dry run and after the release,
    and adds a check with a pull request from a fork.

Changed

  • Breaking for report consumers: the report is now Report v2
    (schema/report.v2.json). schemaVersion is the number 2 (Report v1 used the
    string "1"), every report carries a testIntegrity block (the mode in effect and where it came from, the
    verdict and the signals; mode warn from source default unless set), and the Action's report.json adds a meta block with run
    metadata (event, repository, pull request, head and base commits, run id and attempt). Every Report v1 field
    keeps its name, type and meaning, and no message code changed. report.v1.json has
    additionalProperties: false, so a v2 report does not validate against it: tools that validate the report
    must switch on schemaVersion. The CLI's --format json prints Report v2, and the comment footer reads
    report schema v2. schema/report.v1.json is unchanged.