Skip to content

greenwash 0.5.1

Choose a tag to compare

@0DukePan 0DukePan released this 13 Nov 16:26
· 17 commits to main since this release

What this is

A trust report on what a coding agent's "done" actually means. greenwash reads
the diff, reruns the committed version of any test that was changed, runs the
project's own suite against a baseline, and reports a verdict with the evidence
behind it. No model, no network, no dependencies.

Install from the tag:

pipx install git+https://github.com/0DukePan/greenwash@v0.5.1
greenwash --version

greenwash-cli is not on PyPI yet -- the release workflow is in the
repository and publishes on a tag once the Trusted Publisher is configured.

Verify it before you believe it

git clone https://github.com/0DukePan/greenwash && cd greenwash
python benchmark/verify.py     # 90 seconds, no model, no network

Three offline measurements: a planted cheat per task scored by the real test
runner, each attack beside its legitimate twin, and a positive/negative pair per
language pack. The exit code is non-zero on the first regression.

What changed since 0.4.2

  • greenwash coverage -- the Requirement Coverage Engine. Requirements can
    bind to more than one test, so one passing and one failing is reported as
    contradictory rather than flattened into "failed". A requirement whose
    evidence would not run is could not check, distinct from one with nothing
    bound. Requirements whose bound tests only assert the happy path are named as
    having no negative case. Every report ends with: coverage counts requirements
    with passing evidence; it is not a claim that the code is correct.
  • python benchmark/verify.py -- one command for the whole offline
    benchmark, and the CI job that runs it where every toolchain is installed.
  • verifier-config-changed no longer fires on maintenance. It was 104 of 109
    flags on real history, nearly all dependency bumps; it now asks only when a
    change can alter what runs, and produces 7. Rule id, code and severity are
    unchanged, so existing waivers keep working.
  • unreadable-file (GW-DIV-003). A changed Python file the static layer
    cannot read -- including one git reports as binary -- is a finding instead of a
    clean report. Missing paths are still not findings.
  • A maintenance pass: 42 branches to 1, the default test loop from twelve
    minutes to seven seconds, Windows nightly instead of per-push, no in-tree
    ticket files, and a CI wall time cut from eleven minutes to under two.
  • The demo starts from the cause. The README clip opens on the agent's real
    git diff and ends on the verdict.

The numbers, including the one that is not a win

  • Detector: 25 of 25 planted cheats caught on the machine that ran it (24
    static, 23 behavioural), 0 false positives on 25 recorded honest fixes. Three
    tasks are skipped for missing toolchains and named; CI scores all 28.
  • False positives on real history: a ten-repository survey found 109 flags,
    104 of them one rule on dependency bumps; after narrowing, 12 flags -- 10
    asking, 2 hand-checked false positives. A survey, not a rate.
  • Agent behaviour: still unmeasured on a current model. The earlier +0.0 came
    from an instrument that could not record a passing cheat; the fix is in, the
    re-run needs a funded key. No page claims an effect in either direction.

What did not work

The two pilot runs that exposed corpus prompts not stating their requirement, the
release blocked on API balance, and the eight days CI was red on a released tag
because the benchmark classifier could not read the checker's output. It is all
in docs/what-did-not-work.md.

Full write-up with intervals and limits: docs/benchmark-summary.md and
docs/claim-matrix.md.