Repository navigation
greenproof 0.3.0
Install: uvx greenproof@0.3.0 --version or pipx install greenproof==0.3.0 (PyPI).
Measured against 0.2.0 on the same inputs (MEASUREMENTS.md): of 684 tamper probes, 638 block
and 46 warn (0.2.0: 187 and 42); of 450 honest probes, 2 block (0.2.0: 87). On real history,
75 of 698 test-touching commits block (0.2.0: 70), 21 of them false blocks (0.2.0: 20).
Added
scripts/release_notes.py X.Y.Z, which prints the GitHub Release notes for a version: the
install line, that version's CHANGELOG section and the compare link from the previous
release.tests/test_release_notes.pyruns it against the real CHANGELOG.- The release workflow checks that CHANGELOG.md has a section for the tag before it builds
anything for PyPI, and after PyPI accepts the upload it creates the GitHub Release with that
section as the notes and the built files attached. - Link references at the end of the CHANGELOG for each version and for
[Unreleased]. - A report note on the default branch with no upstream: only uncommitted changes are checked,
and--base REFincludes commits.bench/replay.pytakes--select runner(commits that
touch CI, runner configs,package.json, Makefiles or shell scripts) and--command CMD. runner.tamperedreads more of what chooses which tests run. tox and nox: thecommandsof
each tox environment and each nox session are read, and a-prefix on a check
(commands = - pytest),ignore_errorsorignore_outcomeon an environment, a nox
success_codeswidened ([0, 1, 5],range(10)), asession.runinside atrywhose
exceptswallows it, a check dropped (ruff check .becameecho skipped) and a nox
session.runpath narrowed (teststotests/test_shop.py) block.{envpython},
{toxinidir}and{[section]key}are normalised first, sopython -m pytestforpytest
passes, and a workflow step that runstoxornoxis judged the same way. A baretoxor
noxis judged by what its default environments or sessions run, so a shorterenvlistor a
narrower{posargs:...}default blocks. Also read as
selection: Go-tagsdropped or narrowed; jest and vitest path flags (--testPathPatterns,
--dir) and changed-file flags (--changedSince,--onlyChanged,--lastCommit,
--changed,--related); cargo target kinds (--lib,--bins,--bin,--test,--doc,
--example),--workspacereplaced by-p, and a dropped--featuresor--all-features.
Widening passes.- Monorepo package filters are selection, in
runner.tamperedandci.tampered: turbo
--filter/-F(including!docs), pnpm--filter/-F, nxrun-many--projects/-pand
--exclude, lerna--scope/--ignore, and yarnworkspaces foreach--include/--exclude.
A filter added where the base had none, or one that leaves out a package the base ran, blocks
(turbo run testtoturbo run test --filter=web); a dropped or widened filter passes, and a
rename or a glob (webtoapp-*) warns. Inrunner.tampered,nx run-manybecoming
nx affectedwarns. - A test command that names exact test files covering every file the base path selected
(pytest .topytest test_shop.pywhen that is the only test file) is a warn in
runner.tampered, in any directory, and says that test files added later will not run. Fewer
files, or a test file added in the same change, still blocks. runner.tamperedreads these as selection: pytestpython_files,python_classes,
python_functions,testpaths,norecursedirs,--doctest-modulesand--doctest-glob,-o
and--override-iniof any of them, and the keys of jest (roots,rootDir,
modulePathIgnorePatterns,testRegex), vitest (dir) and mocha (spec,ignore,
recursive) that choose or ignore test files. Each pytest run is judged from the config's
addopts, thenPYTEST_ADDOPTS, then the run's own flags (a command-line-obeats
addopts), following the run throughmake(-Cincluded),npm runand shell scripts. A
flag moved betweenaddoptsand the command line passes, and one dropped from a command line
blocks. With more than 8 different pytest runs the runs are not compared one by one: the
configuration alone is judged, and a change to the runs warns.- Makefile reading follows
includefiles (literal paths, simple variables,$(CURDIR)and
$(dir $(lastword $(MAKEFILE_LIST)))),definemacros used through$(call ...),
conditionals (ifeq,ifdefand the like, read as alternative worlds, so an honest edit in
one branch passes and a swallow in any branch blocks), double-colon, pattern and
multi-target rules, and aGNUmakefilethat make reads beforeMakefile. - Makefile recipes are judged in the directory they run in (
cd dir && ...,make -C dir),
target-specific variables are read in one pass, and a Makefile read only in part still judges
the goal's recipe.set -o pipefailkeeps a pipeline's status whenSHELLis bash, a line
that masks a check is credited when another line of the target runs the same check unmasked,
and a touched line with$(call ...)or$(eval ...)warns, once per target. ci.tamperedjudges abranchesorbranches-ignorefilter added underpush: it passes
whilepull_requeststill runs on every branch, and warns when it takes the default branch
(mainormaster) offpush, or whenpull_requestis itself filtered or gone from the
workflow (a droppedpull_requestalso blocks, as before). It also reads kept exit statuses:
rc=$?; ...; exit 0swallows the check,exit $rckeeps it.
Changed
- Base resolution: on the default branch the base is the merge base with the upstream, so
commits not yet pushed are checked with the uncommitted changes. With no upstream, or HEAD
equal to it (CI on a push), only uncommitted changes are compared. An upstream that shares
no history with HEAD (a shallow clone) is exit 4 with thefetch-depth: 0hint, not a
silent fall back toHEAD. - Python test analysis:
tests.weakened(block) now fires for a test removed by a laterdef,
delor rebinding of its name, a swallowing decorator added to a test,assert [x]or a
non-empty string or tuple,x == x, and an exact numeric check loosened to apytest.approx,
math.iscloseorassertAlmostEqualband of 1% of the expected value or more. A bare
pytest.approx(99), a tight tolerance and a retry or cleanup decorator that re-raises pass. - A comparison with a call (
f(x) == f(x),get_settings() is get_settings()) is a real
determinism or singleton check and passes, unless it replaces a comparison of that call in
the same test (assert refund(100) == 99to== refund(100)), which blocks.delof a base
class that tests subclass removes no tests. - Helper credit follows imports of test-path modules and of test-support modules (those that
importpytest,unittestorhypothesis, or sit undertesting,test_utilsor
pytest_plugin), up to 8 calls deep; application modules earn nothing. A helper or
conftest.pythat loses checks re-analyses the unchanged tests that use it (at most 300
files read and 200 compared, with no note at the cap); one that loses none does not. - Table folds: folding several tests into one table-driven test (a parametrized Python test,
it.each, a Go table, Rust#[case]) keeps itstests.replacedwarn, which 0.2.0 blocked,
only when the table runs, no row leaves early and the row values reach an assertion. It blocks
when an assertion sits inside atry/exceptortry/catchthat swallows the failure, when
it isassert a or bwithadepending on a row, or when a row carries askip,skipifor
xfailmark of a kind the folded tests did not already carry. A Pythoncase _or a Rust
catch-all arm that closes amatch,expect(() => f(row)).toThrow()and.rejects.*count as
assertions, and a keptxfailrow passes. A fold body past the reading budget keeps the warn
and says it is too large to check. - Python warnings:
tests.expected_changedalso fires when a@pytest.fixturereturns a
different literal, andtests.self_mockedwhen an added or changed autouse fixture patches
a callable the tests in its scope assert on. - Go, JavaScript and Rust test analysis:
tests.weakened(block) now fires for a top-level
barereturn,if (true) return;orx || trueguard before the checks, checks under
if falseor&& false,expect(X).toBe(X)self-equality of a call-free expression, and
fewer rows in anit.each,test.eachordescribe.eachtable. Call self-comparisons follow
the Python rule. Areturnthat is the body of a brace-lessif,else,fororwhile,
return expect(...), Go'sgot != gotand a table with a spread or a non-literal table pass.
Helpers in the same file (JS) or directory (Go) earn credit. - A Go
//go:buildor// +buildconstraint added to an existing_test.gofile that had none
istests.skipped(block); one that differs from the file's own warns. gofmt's migration to
//go:build, go fix's// +buildremoval and a constraint on a new test file give nothing. - Data files: a modified, or renamed with changed content, expected-output file (
.json,
.snap,.golden,.csvand similar) under a test path ortestdata,fixtures,golden
or__snapshots__istests.expected_changed(warn), once per file. Dependency, compose,
Dockerfile*,.pre-commit*,codecov.ymland.env*files and pure renames are skipped. ci.tampered(block) judges CI across all workflow files and looks through local composite
actions, local scripts,bash -cand launchers such asdocker compose runanduv run.
It blocks a swallowed check (|| true,|| echo,set +e,; exit 0under a shell with no
errexit),continue-on-error: trueorif: falseon a check,--collect-only,
--passWithNoTests,exit 0before the first check, a selection flag (-k,-m,-run),
fewer events, apathsfilter that hides a test or source file, and a changed greenproof
step orwith:input. Only GitHubrun:steps get the flag checks; other CI files get the
swallow check.- A workflow keeps its greenproof step only if it keeps an automatic code trigger (
push,
pull_request,pull_request_targetormerge_group) that runs on every ref; moving the step
to a manual, scheduled or branch-filtered workflow blocks. Check kinds are per tool (ruffand
flake8differ), so swapping one for the other blocks; an unclassifiable replacement (poe test) warns only in the same file and job, and fewer steps of a surviving kind warn. ci.tamperedpasses a split into a reusable workflow, apathsfilter that only excludes docs
or meta files,rm -rf build || true,continue-on-erroron checkout, cache or upload steps,
andpytest || pytest,pytest || pytest --lf(a retry, not a swallow). Narrowed
branches,tagsortypesfilters (not apushbranchesorbranches-ignorefilter that
leaves the default branch alone whilepull_requeststays open), a matrix-chosen flag,continue-on-error: ${{ ... }}, a mocha
selection change and a ref-only greenproof bump warn. Anif: ''is read as false, and
if: "false"blocks.runner.tampered(block) reads swallows in Makefiles,package.jsonscripts and the files a
policy command runs (deleted, emptied, no-op, swallow, selection flag, earlyexit 0). A Go
TestMainthat skipsm.Run()or always exits 0 blocks, a conditional exit warns. Fewer
steps count checks only, and a script that stops running a kind of check the base ran (jest
becameeslint) blocks, with lint tools as one kind. Makefile variables are expanded, and a
target named like an existing path and not in.PHONYruns nothing.- Flags in
runner.tamperedbelong to their tool (-kis pytest's, mocha's-tis a timeout).
-p no:<plugin>is judged by the plugin:cacheprovider,randomlyandxdistpass,
python,doctest,covandasyncioblock,warningsblocks where the base turns warnings
into errors, unknown plugins warn, and--disable-plugin-autoloadwarns. A nested
package.jsonno policy command reaches warns; with a workspace, filter,cdorlerna
in the policy, it blocks. - The shared shell reader (
rules/shell.py) counts only a known test runner, linter, type
checker or policy command in command position as a check, sopip install pytest-xdist
is not one. Heredocs are data, and only swallows the base lacked are reported. - Performance: git blobs are read through one
git cat-file --batchprocess, workflows are
parsed once, and swallow detection,pathsglobs andpnpm run '/regex/'run in linear time.
The shell reader, the tox and nox readers, the monorepo filter reader and the comparison of
selected paths also take time linear in their input.
Fixed
- A
pnpm run '/regex/'pattern from the candidate could hang the gate (/^(a+)+$/); it is now
matched by an automaton with a work budget. Names are no longer cut at 200 characters, which
let/^[a-z]*$/match a long name ending in!. - An
if:wrapped in redundant parentheses is read as the expression inside, however many
pairs wrap it, and the wrapping does not count toward depth. A new or changedif:with more
than 32 levels of real&&/||nesting on a job or step that runs a check is a warn, "has an
if:nested too deeply to read". - Dropping a workspace member from
pnpm-workspace.yamlorworkspacesmade its neutered
package.jsona warn; it now blocks. - A huge numeric literal or a very deep expression in a test no longer crashes the check, and
Rust character literals such as'}'no longer break brace matching. - False blocks cleared: a
|| trueafter a command that is not a check (rm -rf build,
pip install x),-p no:randomlyand other plugins that only reorder or report, and a
nestedpackage.jsonthat no policy command reaches (now a warn).
Full changelog: v0.2.0...v0.3.0