Skip to content

v0.2.0 — measurement and distro awareness

Latest
Latest

Choose a tag to compare

View all tags
@acceptha acceptha released this 26 Oct 10:14
· 6 commits to main since this release
v0.2.0
0c71c00

Measurement and distro awareness: a held-out benchmark set, a benchmark of the gate itself,
per-model gate acceptance, and backport status from the distributions carried into the guide
and enforced by two new gate rules.

Added

  • Backport status (bcpg.match.distro, bcpg scan --distro R --pkg-version V): each
    upstream CVE is open / fixed only in Ubuntu Pro / fixed by a backport / has no distro record,
    from the Debian and Ubuntu records in OSV for the installed package version. Shown in
    report.md (section and column), report.json and the trace. Rules gain distro_package.
  • Holdout benchmark set: 33 libraries from Debian 12, Ubuntu 22.04 and 24.04, extracted
    after the rules were frozen (extract.py --set holdout). Frozen rules: 27/33 exact versions;
    after two rule fixes 29/33; the other four files carry no version string.
  • bcpg gate-bench: false rejections of correct advice (0/540, 18 phrasings) and detection
    of 22 injected fault kinds (572/572), plus two out-of-scope semantic faults (0/54, reported).
  • bcpg model-bench: per-model gate acceptance per batch, rejections by rule letter,
    tokens and time; results merge by model in bench/model_results.md.
  • make fuzz: libFuzzer + ASan/UBSan harness for the C ELF parser in Docker
    (10 minutes, 2.38M inputs, no crash).
  • README.ko.md and docs/design.ko.md (public design-decision record).
  • docs/demo.gif (recorded with vhs docs/demo.tape, offline, no LLM).
  • Distro-aware guide: with --distro, each CVE's backport status goes into the facts (with
    a per-CVE must_say line), the prompt and the template; validator rule (k) rejects an
    action for a CVE the installed package already fixes unless it says so, and a Pro-only fix
    unless it says Pro. Package and release versions (~18.04.23, Ubuntu 20.04) are no longer
    read as invented component versions. Sample: docs/sample/zlib-1.2.11-debian10/.
  • Model benchmark row for qwen2.5 7B (4/9 batches accepted vs 6/9 for exaone3.5 2.4B).
  • make lint; CI also runs ruff format --check.
  • Validator rule (l): input field names (facts.…, allowed_refs, …) in the report text;
    rule (k) also rejects a "not fixed yet" phrase next to a CVE the distribution fixes or has no
    record of. Both from a qwen2.5 7B guide that passed the earlier rules.

Changed

  • Offline OSV misses are logged as one line per run instead of one per CVE (47 lines for a
    holdout libcurl scan before).

Fixed

  • report.md, report.json and trace.json are written with LF on Windows too.
  • Validator rule (i) rejected correct advice that listed protocol versions ("TLS 1.2/1.3");
    found by gate-bench.
  • Rules: libxml2's integer version glued to preceding bytes (::20914, Debian 12); zlib builds
    without copyright banners (bare ZLIB_VERSION, trusted inside libz.so only).

Key numbers: holdout set 29/33 exact versions (27/33 with the frozen rules) · gate 0/540 false rejections, 572/572 targeted faults detected · local LLMs on CPU: exaone3.5 2.4B 6/9 batches accepted, qwen2.5 7B 4/9.

Full changelog: CHANGELOG.md

Assets 2
Loading