signetry-eval v0.3.0 — the Agent Governance Leaderboard

Added — the Agent Governance Leaderboard

Detection is table stakes. The axis nobody publishes is governance: when the
repository itself is hostile, does the agent's change still get admitted, and what does
the defense cost in benign work? docs/LEADERBOARD.md publishes
both axes on one page and takes third-party submissions, so the governance axis can
become a real comparison instead of a self-report.

• signetry_eval/leaderboard.py renders the page and enforces three rules in code
  rather than by good intentions: an unmeasured number is — and never 0%;
  reproduced and self-reported rows never share a table; every rate is printed next to
  its denominator. A submitted rate with no denominator is refused and shown as
  unmeasured, with the reason listed on the page.
• signetry-eval leaderboard (--with-detection, --json, --entries) — a new
  subcommand. Exits non-zero if any governed defense failed.
• leaderboard/entries/ takes one JSON file per system, schema documented in its
  README. The ungoverned baseline ships as its own
  visible row rather than being implied by a column heading.
• docs/SUBMITTING.md — how to submit an attack that beats the
  governed pipeline, and how to submit a system, including one that beats Signetry.
• .github/workflows/leaderboard.yml regenerates the page weekly, on release, and when
  an entry or scenario changes. benchmark.yml moved onto the same schedule: both pages
  were previously release-only, so published numbers could be months stale while still
  reading as current. Each workflow owns exactly one page, so every "Generated by …"
  header names the workflow that actually wrote it.

Fixed — a competitor was scored on cases it was never run on

The corpus benchmark charged a replayed scanner with a miss for any case absent from
its capture. Captures are taken at a point in time and this corpus grows, so the 8 cases
added in the 52 → 60 expansion were counted as failures for
claude-code-security-review — 5 of them vulnerable. Its published recall read 81%
(38/47) when the honest figure over the cases it was actually given is 90%
(38/42).

That is the same defect this suite calls out everywhere else — a score on evidence that
does not exist — except pointed outward at a named tool, and it inflated our lead by
about nine points.

• A case absent from a scanner's capture is now covered=False and excluded from its
  recall, false-positive count, and per-family/per-language breakdowns. An entry that is
  present but lists no findings is still a genuine miss; only "never run" is excused.
• Every table now prints Cases scored per scanner, and the excluded case ids are
  listed under Notes — excluding them silently would be its own dishonesty, since the
  two scanners are no longer scored over the same set.
• CorpusScore.recall returns None rather than 0.0 when nothing was measured, and
  the --min-recall gate now fails on an unmeasured recall instead of passing it.

Fixed — unmeasured per-category metrics rendered as 0%

Report.by_category() reported 0 for metrics with no evidence behind them: an
attack-only category showed Utility 0% (reading as "Signetry destroyed utility here")
and the utility category showed ASR 0% over zero attacks. Both are now None, render
as —, and serialize as null. A measured zero is still reported as 0% — the rule
is no unearned numbers, not no zeros.

Changed — Signetry is now open core; this repo is Apache-2.0

• An Apache-2.0 LICENSE file is now present, replacing the previous
  "All Rights Reserved" terms, as part of Signetry's
  open-core model. The
  engine (Signetry/core) is source-available under
  BUSL-1.1 and converts to Apache-2.0 on 2030-08-31.
• This repository has no strings deliberately. A benchmark nobody can freely run,
  audit, and reproduce is worthless as evidence, so the eval suite carries the most
  permissive licence of anything in the platform — fork it, re-run it, publish results
  that disagree with ours.
• pyproject.toml declares license = "Apache-2.0" and the OSI Apache classifier,
  replacing Proprietary — All Rights Reserved.
• The all-rights-reserved framing is gone from README.md, CONTRIBUTING.md,
  CLA.md, CONTRIBUTORS.md, and the CLA workflow's PR comment.
• The CLA is kept, and its fallback licence grant is now non-exclusive so a
  contributor never loses the right to use their own contribution. See
  CLA.md §2–3.

Added — community health files

• SECURITY.md, CODE_OF_CONDUCT.md (Contributor Covenant v2.1), and GitHub issue
  templates.

Added — OWASP breadth in the detection corpus (52 → 60 cases)

• XXE (CWE-611) in Java and PHP — LANG-53, LANG-54 (eval#11).
• Path traversal (CWE-22) in Go and Java — LANG-56, LANG-57 (eval#12).
• SSRF (CWE-918) in Go — LANG-59 (eval#29), giving the class a second
  language alongside Python.
• Three SAFE decoys probing the precision distinctions these rules must make:
  default-safe PHP XML parsing (entities are off by default on PHP 8+), a constant
  filesystem path, and a constant host with a user-supplied query string
  (LANG-55, LANG-58, LANG-60).
• Two pinned real-repo cases — OWASP WebGoat (Java) and OWASP RailsGoat
  (Ruby), the first JVM/Ruby targets here; every prior case is Python or
  JavaScript (eval#13).

The LANG-60 decoy earned its keep immediately: it caught a false positive in
signetry-core's new Go SSRF rule, fixed in Signetry/core#97 before this landed.

Fixed — pinned real-repo cases were not actually pinned

• scan_real_repo cloned with --depth 1 and then ran git checkout <sha> with
  check=False. On a shallow clone the object is absent, so the checkout failed
  (fatal: unable to read tree), the failure was swallowed, and the scan silently
  ran against the default-branch tip — a case documented as "pinned for
  reproducibility" was not pinned. Now fetches the specific object first, and if
  pinning genuinely cannot be honoured it says so in the result note rather than
  reporting an unpinned scan as pinned.

Changed

• Pin signetry-core at v0.7.0; the corpus additions above depend on its new
  Go SSRF / Go+Java path-traversal / PHP XXE rules.

Changed — Signetry rename (breaking)

• Distribution signetry-eval and import package signetry_eval. The console
  command is signetry-eval.
• Environment variables use SIGNETRY_*; the config directory is .signetry/.
  Product/brand prose updated to Signetry.
• Core dependency pinned: signetry-core @ git+https://github.com/Signetry/core@v0.6.0
  (was signetry-core @ ...@v0.5.4). Imports use signetry_core.
• No backward-compatibility fallbacks are provided.

signetry-eval v0.2.3

Signetry rename release.

Install:

pip install "signetry-eval @ git+https://github.com/Signetry/eval@v0.2.3"

umbra-eval v0.2.2

Changed

• Install umbra-core from its source repository (git+https://github.com/bkd-dotcom/umbra-core@v0.5.4)
  instead of PyPI — umbra-core is All Rights Reserved and no longer distributed on
  PyPI. Enables tool.hatch.metadata.allow-direct-references.

umbra-eval v0.2.1 — README benchmark docs + umbra-core>=0.5.0

• README now documents the detection head-to-head benchmark (52 cases / 7 languages; umbra-core 100% recall / 0 FP vs Claude Opus 4.8 90%) + the corpus/realrepo commands.
• Fix: dependency floor raised to umbra-core>=0.5.0 so a fresh install always has the detection engine.
• Honest note: any false positive comes from the optional Semgrep layer, not the deterministic engine (which is 0-FP on the corpus).

umbra-eval v0.2.0 — detection benchmark

First tagged release since the head-to-head detection benchmark landed.

• 52-case, 7-language public corpus (public/OWASP, academic/CWE, crafted, hard cross-file taint, multilang, cross-file-lang) with cited provenance + safe decoys.
• umbra-eval corpus scores Umbra vs competitor scanners (recall, false positives, by-language). --semgrep enables the optional layer.
• Committed result: umbra-core 100% recall / 0 false positives vs Claude Opus 4.8 90% — deterministic, offline, free.
• Plus the ASR / utility-under-defense adversarial suite (umbra-eval run).