Skip to content

CodeWitness 1.0.24

Choose a tag to compare

@github-actions github-actions released this 08 Nov 15:39
· 2 commits to main since this release

CodeWitness 1.0.24 fixes four defects found by pointing the tool at a tree that is not its own — a polyglot
project with a Python front end and a Rust crate in a subdirectory. A crate whose Cargo.toml sits below the
project root is now found and built instead of reported as "no Cargo.toml was found"; one language's build no
longer blocks requirements in the languages that built; and two Python rules stop reporting method definitions
and docstring prose.

What changed

  • A Cargo manifest below the project root is found, and every Cargo command names it. The adapter asked one
    question — Files.isRegularFile(root.resolve("Cargo.toml")) — and Cargo itself resolves no manifest above its
    working directory, so rmath/Cargo.toml was invisible while the crate's .rs files were still detected: a
    Rust project was reported and then refused. Detection now searches a bounded depth (6) below the root, skipping
    target, .git, .codewitness, node_modules, vendor and .cargo — and also fixtures/testdata, because
    a repository's sample crates are inputs to its tests, not code the run verifies. The shallowest manifest wins,
    deterministically (depth, then path). Its directory becomes the build root, and crate name, edition, toolchain,
    src/ and tests/ are read from there. Build, tests, coverage, mutation, generated tests and the dependency
    inventory all run against it via --manifest-path, so a report's commands stay runnable from the project root.
    A tree with several manifests and none at the root is stated rather than glossed over: the report names the
    crate that was built and the manifests that were not part of the run's evidence.

  • One language's build no longer blocks requirements in the languages that built. blockedFor blocked per
    language only when it could attribute a requirement to one — attribution being a requirement id in a source
    file or a declared paths/tests. A requirement written in prose has no language, and the fallback then
    consulted the merged build, so on a tree where Python built and Rust did not, every requirement came back
    BLOCKED — including "The Python front-end suite passes" — while the report's own limitation text said the
    opposite. Such a requirement is now blocked only when nothing in the tree built; with something built it is
    judged on the evidence that exists, and its verdict names the languages it could not draw on and says how to
    have it judged per language next time.

  • A build that was never attempted is no longer merged into a failed build. mergeBuilds forced
    attempted(true) on the merged result and counted a language whose build was never attempted as a failure,
    which is how one run reported build failed with exit code -1 … language(s) whose build failed: [rust] while
    Rust's own per-language note said build not attempted (no Cargo.toml was found). languageSummaries
    compounded it by printing buildAttempted: true for a language whose result said attempted() == false. The
    merged result now reports attempted as any-language-attempted and success over the languages actually
    attempted, each language summary carries its own flag, and the limitation text separates "failed to build"
    from "no build was attempted".

  • CW-PY-SEC-002 no longer fires on a definition named eval. The rule was
    (?<![\w.])eval\s*\(|(?<![\w.])exec\s*\( matched against a raw line: the lookbehind spared self.eval(, but
    in def eval( the character before the name is a space, so a method definition read exactly like a call to
    the builtin. On one project that was nine HIGH security findings and not one call. The name a
    def/async def/class line defines is now blanked before the rule table sees it.

  • CW-PY-BUG-002 no longer fires on docstring prose, and no longer treats is True/is False as defects.
    Three of its five findings on that project were English sentences inside docstrings ("1.0 is 1 and 2.5 is
    5/2", "…a negative is 0") and two were x is False checks. Two separate mistakes: the analyzer matched every
    line of a file with no notion of strings, comments or docstrings — it was the only one of the sixteen that
    did; the others all have a codeOnly view — and the pattern listed True/False as identity-comparison
    literals although is True / is False / is None are the form PEP 8 recommends, and the runtime
    SyntaxWarning it cites applies to numbers and strings. The Python analyzer now tracks string literals and
    comments line by line, including triple-quoted strings across lines, and discards any match that begins
    inside one — for every rule, not just this one. The rule keeps only the literals Python actually warns about,
    and its message says that is None / is True / is False are not reported.

What this does not fix

  • One crate per run, when the tree holds several manifests and none is at the project root. A Cargo
    workspace declared at the root is built as a whole, as Cargo builds it; a tree with sibling crates is not.
    docs/status.md states the limit and the report names the crates that were not built.

  • Requirement → language attribution is unchanged. It still comes from a requirement id appearing in a
    source file, or from a declared paths/tests. The new behaviour makes an unattributed requirement honest —
    it is no longer blocked by a language that is not its own — but it cannot make it attributable. A requirement
    that names its files gets a per-language verdict; one that does not is judged on whatever evidence exists,
    and its verdict says so and says how to fix the mapping.

  • The run-level build summary is still one line for the whole run. It succeeds only when every build that ran
    succeeded. What changed is that a language nobody attempted to build no longer counts as a failure; the
    per-language record, with its own attempted/succeeded pair, is in the language summaries and the evidence
    ledger.

  • The identity rule is now narrower than a style linter. is True / is False are not reported at all. A
    project that wants them flagged should use a linter; CodeWitness reports what Python warns about.

  • The Python rule table is still a line-oriented heuristic table. The string and comment tracking is per
    line and does not attempt a full parse, so a pathological literal — a raw string ending in a backslash, say —
    can still confuse the spans. Every rule in the table gained the same view, so a wrong span suppresses or
    admits a finding rather than crashing the phase.

  • Python existing-test execution still depends on the project interpreter. Whether pytest and coverage.py
    are importable by the interpreter CodeWitness picks decides whether the suite and coverage can be measured;
    that is unchanged and reported per run.

Verified for this release

  • Full reactor suite: 1,784 tests, 0 failures, 0 errors, 0 skipped — 21 more than 1.0.23's 1,763: 14 for the
    Rust manifest work (NestedCrateTest, RustMutationProfileTest, four RustProjectDetectorTest cases) and 7
    for this release's engine and Python rules.
  • Every new guard was proved to fire, each against the pre-fix code and each restored afterwards:
    • Against the pre-fix detector, eight of the new Rust cases fail — including the exact message the defect
      produced, no Cargo.toml was found, so mutants cannot be compiled;
    • against the pre-fix engine, the aggregate-gate case fails with R-001 … status=BLOCKED and the
      never-attempted case fails on both buildAttempted and the merged build's success;
    • against the pre-fix Python rules, three cases fail with the reported findings reproduced exactly —
      def eval(self, text, steps=False): and async def exec(self, text): reported as HIGH security findings,
      if width is False: reported as a defect, and prose in a trailing comment reported twice.
  • Measured on a copy of the project that produced the reports (never its working tree): CW-PY-SEC-002
    went from 9 HIGH findings to 0 and CW-PY-BUG-002 from 5 MEDIUM to 0, while CW-PY-REL-003 (62) and
    CW-PY-TEST-001 (2) were unchanged — the rules that should still fire still do, and a real eval() call,
    x is "literal" and a credential literal are still reported.
  • End to end on a polyglot fixture (Python that builds, a nested Rust crate that does not): before, every
    requirement BLOCKED and the run BLOCKED; after, R-001 VERIFIED from an executed scenario while Rust
    failed, R-002 UNVERIFIED, run PARTIALLY_VERIFIED, exit 0, and the limitation naming Rust as the language
    whose build failed.
  • Self-check against the built jar (./codewitness verify . --requirements selfcheck/requirements.yaml --config selfcheck/codewitness.yaml --no-mutation --max-duration 30m): all seven ground rules VERIFIED,
    1,784 tests executed and passed, 0 findings — five suppressed with their reasons — coverage 71.3% line
    (25,775/36,141) and 59.6% branch, 30 resolved dependencies with 0 advisories from OSV, and exit 0 in
    1 m 50 s. Run CW-2026-11-08-BYFSS. selfcheck/README.md quotes this run.
  • ./codewitness version prints codewitness 1.0.24.
  • Rust end-to-end suite (-Pe2e): 2/2.

Process note

The aggregate-gate defect is worth recording because of how it was found: not by a failing test but by a
report contradicting itself. Run CW-2026-11-08-W1KWN said every requirement was BLOCKED and, in the same
limitation list, that "requirements written in a language that did build were still verified". Both sentences
were produced by the same run, and the wrong one was in the verdicts. A sentence in a report is a claim, and
a claim that is not tested is a claim that drifts
: blockedFor implemented per-language blocking and the
limitation text described it, but nothing exercised the case where a requirement has no language at all, so the
two halves were free to disagree.

The two Python rules failed the same way from the other direction. The rule table had tests, but every one of
them fed it clean code: a call, a comparison against a string. Neither rule was ever run against a definition
named eval or a docstring that uses the word "is", which is what a real project is full of. The precision
work now starts from the two shapes a real file has that a fixture does not — a method whose name is a builtin's,
and prose that looks like code — and the analyzer gained the string/comment view the other fifteen already had.

Compatibility

  • No CLI command, flag, exit code contract, report key, SARIF profile, configuration key or adapter-SPI change.
    Every codewitness.yaml key and report field behaves as it did.
  • Behaviour changes, all in the direction of fewer false findings and fewer false BLOCKED verdicts:
    • a polyglot run may now be COMPLETED/PARTIALLY_VERIFIED where it was BLOCKED, and the exit code follows;
    • CW-PY-SEC-002 and CW-PY-BUG-002 produce fewer findings; finding fingerprints are unchanged in shape, so
      baselines and suppressions keyed on them keep working, and a previously suppressed false positive simply
      stops being reported;
    • reports from a nested crate record --manifest-path in their commands and report the crate directory as the
      build root;
    • buildAttempted in a language summary now says false for a language nobody built.