cvehound-spatch 1.3.3.post1

Coccinelle 1.3.3, now able to say whether a scan finished. Handed a directory,
stock spatch skips any file it raised on and still exits 0, so an empty answer
cannot tell a clean tree from a file that was never scanned. This build reports
which it was, keeps a damaged or foreign AST-cache entry from costing a file,
and lets a consumer detect all of it with spatch --features.

Every scan ends with a summary

Every scan prints one line on stderr as its last line, even under --very-quiet:

SPATCH-SUMMARY groups=<n> handled=<n> skipped=<n> dropped=<n> timeouts=<n> unreadable=<n>

groups = handled + skipped + dropped. The key order is fixed, and new keys are
only ever appended. Each file the scan lost gets its own line, with the file
last so a path with spaces survives:

SPATCH-DROP kind=<timeout|exn|unreadable> reason=<...> file=<path>

This includes unreadable files, which used to leave no trace at all. A missing
summary means the scan did not finish. The exit code does not change, so a
harness that needs to know whether a scan was complete reads the summary.

The AST cache no longer costs a file

• cache-selfheal: a --cache-prefix entry that fails to read is now a
  cache miss: the file is parsed again and the entry rewritten. Before, the
  read error dropped the file on every scan sharing that cache, with exit 0,
  until the source changed.
• cache-buildid: spatch --version now ends with a Build: <commit>
  line. The cache key now also includes the build, the OCaml version and the
  parse flags, so an entry written by another build or under other options is
  parsed again instead of read. Entries left by older wheels miss once and are
  overwritten.

--timeout no longer cuts a directory scan short

The CPU timer behind --timeout was never stopped after a file group finished.
With --timeout 5 and a file that took 1 s, the remaining 4 s could run out
later in the prefilter, which does not catch a timeout. The whole scan then
exited 124 without looking at the rest of the tree and without naming a file.
The timer is now stopped when each group finishes.

Zygote replies

• If a request's environment contains COCCI_ZYGOTE_PROTO=2, the reply also
  says how many bytes the child left in each capture file and, for a signal,
  gives its name. A capture that was cut short is now detectable:

  exit <n> stdout=<bytes> stderr=<bytes>
  signal <n> name=<SIGNAME> stdout=<bytes> stderr=<bytes>
  

  A size is -1 when the child died before creating that file.

• The v1 reply is unchanged except for the signal number, which is now the
  host's own: signal 9 for SIGKILL. It used to be OCaml's negative constant,
  signal -7, which a client that flipped the sign read as exit code 7.

Feature detection

spatch --features prints one token per line; older builds reject it with exit
2. FEATURES is still measured from the built binary, now through that option:

>>> sorted(cvehound_spatch.FEATURES)
['cache-buildid', 'cache-selfheal', 'exit124', 'features', 'shared-cache',
 'summary', 'zygote', 'zygote-v2']

A consumer checks for a token, e.g. 'summary' in getattr(cvehound_spatch, 'FEATURES', frozenset()), and still works with any older wheel.

Wheels for Linux x86_64/aarch64 (manylinux_2_28) and macOS arm64
(macosx_11_0_arm64). Needs pip 20.3 or newer to install, and diff on the
system at run time.

pip install 'cvehound[spatch]'

cvehound-spatch 1.3.3

Coccinelle 1.3.3, built the same way as before. The one user-visible consequence
is that this wheel is no longer ahead of a stock 1.3.3 on satLabel: the caching
fix this package carried since 1.3.2 was merged upstream and ships in the release
itself.

satLabel caching is upstream now

coccinelle#417 — the large
regression on rules that wrap a pattern in a function context, 35× on the rules
that trigger it — landed in 1.3.3 verbatim, so the fork branch carries seven
commits on top of upstream instead of eight and the README's list of deviations
shrinks by one. Nothing changes for anyone installing this package; it is simply
no longer the only way to get that fix.

Everything else is unchanged

The rebase moved nothing else in the fork's diff. --zygote, exit 124 on an
engine timeout, the shareable AST cache, the memoised per-file overhead and the
max_cnf atoms-only prefilter fallback
(coccinelle#420, still
carried) are all intact, and so are the build choices: no Python or OCaml
scripting, no PCRE syntax, OCaml 5.3 with flambda -O3 and --without-zstd,
bounds checks kept.

FEATURES is still {'exit124', 'shared-cache', 'zygote'}, so there is nothing
new for a consumer to gate on — cvehound-spatch>=1.3.2.post3 already resolves
to this release.

Wheels for Linux x86_64/aarch64 (manylinux_2_28) and macOS arm64
(macosx_11_0_arm64). Needs pip 20.3 or newer to install, and diff on the
system at run time.

pip install 'cvehound[spatch]'

cvehound-spatch 1.3.2.post3

Same coccinelle 1.3.2 build as 1.3.2.post2 — no patch changes — this release adds a wheel for macOS, so pip install 'cvehound[spatch]' gets a prebuilt binary there too instead of falling back to PATH.

macOS arm64 wheels

Wheels now ship for macOS arm64 (macosx_11_0_arm64) alongside Linux x86_64/aarch64 (manylinux_2_28). There is no container to build portably against on macOS, so build.sh runs directly on the host and Homebrew-installs whatever autogen and opam need; it rejects the build on the same grounds as Linux — anything linked beyond the platform's own libc, or Python support — and on macOS it additionally rejects a binary whose LC_BUILD_VERSION disagrees with the deployment target the wheel tag promises, or whose ad-hoc signature doesn't verify. CI verifies each platform's wheel against its own runner; macOS has no Linux-style "older/newer than the build image" pair, since the wheel's floor there is a deployment target, not a userland version that could disagree.

OCaml built --without-zstd

From OCaml 5.1 the runtime links libzstd wherever configure finds it, and a wheel can't ship a Homebrew dylib alongside it — this surfaced as /opt/homebrew/opt/zstd in otool -L. The manylinux image happens to carry no zstd headers, so the Linux build was getting the same result by luck; asking for ocaml-option-no-compression explicitly makes the invariant hold on both platforms. Nothing is given up: compressed marshalling was opt-in, and this build ships none of the artifacts zstd would have shrunk.

Wheels for Linux x86_64/aarch64 (manylinux_2_28) and macOS arm64 (macosx_11_0_arm64). Needs pip 20.3 or newer to install, and diff on the system at run time.

pip install 'cvehound[spatch]'

cvehound-spatch 1.3.2.post2

Rebuilds the bundle from the cvehound branch, which has gained six commits
since 1.3.2.post1. Four of them change how spatch can be driven, and this
release is what makes them reachable — including a way for a caller to find out
that they are there.

spatch can now be run as a server

spatch --zygote reads length-prefixed requests on stdin and forks a fresh
process for each one, after warming standard.iso and standard.h once. A
scan that runs hundreds of rules pays process startup once instead of once per
rule. Every request still runs in its own process, so no engine state carries
from one rule to the next and results are unchanged; measured at roughly 5% of
a CVEhound scan today, and more as the rule set grows.

A fired --timeout is now distinguishable from a crash

It used to escape as an uncaught Common.Timeout, i.e. exit 2, which is also
what any other uncaught exception gives. It now exits 124, the way
timeout(1) does. The exception name stays in the message, so a caller that
already scrapes stderr keeps working.

The AST cache is safe to share

--cache-prefix entries are written through a temporary name and renamed,
value before dependencies, so parallel scans sharing one cache directory cannot
read a half-written entry. standard.h is now part of the cache key. The cache
path also no longer forks /bin/sh to run mkdir -p once per file — which a
cache hit was paying too.

FEATURES, so callers can ask

>>> import cvehound_spatch
>>> cvehound_spatch.FEATURES
frozenset({'exit124', 'shared-cache', 'zygote'})

None of the above can be detected from outside: --zygote is dispatched before
spatch parses its arguments, so it is in no --help output, and the version
banner is identical with and without it. make_wheel.py probes the built
binary at pack time rather than asserting, so a build from a ref that predates
a feature advertises nothing instead of lying. Read it with
getattr(cvehound_spatch, 'FEATURES', frozenset()) and every older wheel
correctly reports that it has none of this.

CVEhound uses exactly that to decide whether to run spatch as a server, so
cvehound[spatch] installs get the faster transport automatically from
this release on, and installs on an older wheel keep the previous behaviour
rather than failing.

Built with flambda

The compiler study was re-run against the real rule corpus rather than a
handful of heavy rules, and flambda -O3 measures 4.5% of CPU (5.3% under
the zygote transport), against the 2-3% that had it rejected the first time.
It costs +11.4 MB of binary, which buys 428 extra minor page faults per exec —
about 0.05 ms. llvm-BOLT was measured again too and is still not shipped; the
reasoning is in the README.

cvehound-spatch 1.3.2.post1

The same coccinelle 1.3.2 build as 1.3.2 — no packaging changes — rebuilt from
evdenis/coccinelle@48bf7b1
to pick up a second carried patch.

Fixed: hitting max_cnf disabled the file prefilter entirely. When the CNF
conversion in interpret_cocci_git_grep exceeded max_cnf it gave up and
returned "no query", which worth_trying reads as "try every file" — the
semantic patch was then run against the whole tree with no prefiltering at all.
It now degrades to a one-clause query over the atoms, which is still a sound
filter because the formula is negation-free. Submitted upstream as
coccinelle#420.

The wheel therefore carries two patches on top of coccinelle 1.3.2: the
satLabel caching fix (coccinelle#417)
that was already in 1.3.2, and this one.

Wheels for Linux x86_64 and aarch64 (manylinux_2_28). Needs pip 20.3 or newer to
install, and diff on the system at run time.

pip install 'cvehound[spatch]'

cvehound-spatch 1.3.2

First release: coccinelle 1.3.2 built without Python or OCaml scripting and without PCRE syntax, on OCaml 5.3, with the satLabel caching fix (coccinelle#417) applied.

Wheels for Linux x86_64 and aarch64 (manylinux_2_28). Needs pip 20.3 or newer to install, and diff on the system at run time.

pip install 'cvehound[spatch]'