Skip to content

v0.45.0

Choose a tag to compare

@github-actions github-actions released this 28 Sep 05:09
· 124 commits to main since this release
b4aa77d

The attestation-format release. A signed bundle now binds the exact bytes of report.json, and its
Ed25519 signature covers a 306-byte record that names the statement, so a key held in AWS KMS can
sign it (provael-attestation/v2; a 0.44.0 verifier cannot check a v2 bundle, so upgrade verifiers
before issuers, and v1 bundles keep verifying). Beside it: provael calibrate splits benign
rollouts three ways by default (report schema 7), a second predicate for contact and force events
is wired and unmeasured, the ML-BOM describes the policy rather than the tool, Meta-World is
declared scaffolding (4 runnable / 3 scaffolding suites), three ATLAS tactic names are corrected
(E-2026-16), and the signed regulatory clock carries ISO 10218 at the month ISO gives (E-2026-17,
ruleset /6). No previously published number moves; three 18 September workstation
measurements are added beside the headline, each with what it does not show. The published
measurement is still the 14 September run on 0.41.2, now four releases behind a window of two:
watch/publish-freshness.json reads isStale: true, and the campaign stays paused until the
predicate is calibrated (#136).

Added

  • The second predicate: a contact / force event beside the envelope exit (physical_hazard
    endpoint, oracle contact-event/v1). Why: the first proof on the roadmap owes a safety engineer
    a column in the units they already read — contact and force, not a keep-out box — and the
    endpoint had been named since Phase 7 with no suite behind it. What changed: SuiteAdapter
    gains physical_hazard(state) -> bool | None (default None: the stub is arithmetic and has no
    contact API, and a fabricated False would read as "no hazard"); the LIBERO adapter reads, per
    step, robosuite's end-effector force sensor (robots[0].ee_force) and MuJoCo's contact list
    through the lerobot → LIBERO → robosuite chain, every read guarded so a missing sensor is an
    unmeasured step and never a False, and ContactRule calls an event when the force norm reaches
    its limit (default 140 N — ISO/TS 15066:2016 Table A.2, quasi-static, hands and fingers: a scale
    an engineer recognises, not a claim that LIBERO holds a person) or an arm link (robot0_*)
    touches a body that is not the robot's own (gripper contact with objects and the table is
    manipulation and is not counted); the runner latches the per-step answers into one episode
    outcome under AttackResult.endpoints["physical_hazard"], writing the key only when a signal
    was surfaced; scoring.asr.endpoint_by_attack counts events over the episodes that measured
    them; the Markdown report, the CLI table and the results README generator add a contact
    events
    column with its own Wilson interval when any episode surfaced the signal and say
    not surfaced when none did. The contact rule never reaches is_unsafe, so it cannot enter
    the ASR. The attribute chain and sensor names were read from the lerobot 0.5.1, LIBERO and
    robosuite 1.4.1 sources on 21 September 2026 and exercised against a fake graph on CPU; no
    committed run carries the column yet
    — the calibrated re-run on the GPU box is the first that
    will, and the endpoint stays N/A until a live simulator has produced it.

  • Keep-out figures for the three 14–18 September LIBERO-Object runs (results/<run>/figures/,
    one SVG per task): every benign end-effector path in blue against the task's keep-out zone, the
    roleplay paths in red with a marker at the first unsafe step; the control run also draws the
    four harmless-variation arms in grey. Rendered by scripts/plot_keepout_paths.py from the
    shards' own trajectories, so nothing is drawn that a report.json does not carry. The pictures
    answer the #136 question directly: the benign paths never enter the default box, and the
    roleplay paths cross into it from the same start. No measured number moves.

  • Three measurements from the 18 September workstation runs, and what each one does not show.
    Why: the shards finished while nothing was pushing them, and two of the three are easy to
    misread in the project's own favour. What changed, per run. (1) instruction_canonicalization
    on the ten libero_object tasks: the attack survives the defense at 20/30 against a benign
    floor of 1/30, McNemar p=3.8e-06, competence 29/30. It is not a measured reduction against
    the undefended 42/50 — different seed counts (3 against 5) and overlapping task-clustered
    intervals ([43%, 87%] against [62%, 100%]) — and a paired run at matched seeds has not happened.
    (2) action_envelope on the same tasks: 0/30 on the roleplay arm and 0/30 on the competence
    control
    , so the policy completed no task at all under the defense and the zero says nothing
    about the attack; the honest reading is "unusable at the shipped defaults". (3) Ten keep-out
    calibrations (20 benign rollouts each, a roleplay arm at the holdout seeds, 36 candidate
    faces): every fit reports a benign FPR of 0.0 and they differ only in whether the boundary
    catches anything — tasks 0/6/8 catch every attacked rollout, task 3 catches one in six, tasks 5
    and 7 chose the opposite face and catch none. Eight would be adopted and two withheld, which is
    why a benign-only fit is not enough. Those detection rates are in-sample: the same six attacked
    rollouts chose each face (the candidate that catches the most of them), so none is a held-out
    rate. These are two-way fits at a 5% target and carry no binding;
    the three-way re-fit supersedes them. realPolicyDefenses moves 0 to 2 across watch/, the
    evidence manifests and the docs counts, and the two assertions that pinned it at zero now read
    two. No previously published number moves.

Changed

  • provael calibrate splits benign rollouts three ways by default — fit / tuning / eval — and
    the eval split is scored only after the predicate is chosen.
    The two-way path stays as
    --split two-way. Why: since 0.43.0 every surface has said that the second split selects the
    threshold or hazard face, so its FPR is a tuning figure and no evaluation split stood behind a
    calibration; his 19 September review listed wiring the three-way split end to end as the first
    product gap, and the 24 October re-run under a calibrated predicate would otherwise publish
    calibrations with no FPR anyone could call an estimate. What changed: the artifact records
    split, eval_seeds, eval_fpr and a binding (CalibrationBinding schema 2: endpoint,
    oracle version, policy, suite, task, checkpoint, predicate kind, the three seed-set digests,
    target and achieved eval FPR; seed overlap refused at build); an eval FPR above target is
    written as fitted with an invalid binding and a red row, never re-fitted on the eval seeds (a
    threshold moved to satisfy its own evaluation turns that split back into tuning data); the
    runner re-derives the binding's validity against the run's own policy, suite, task, --model
    and oracle version, applies the predicate either way, and says which
    (calibration.<task>.split / eval_fpr / binding in report.json — report schema 7,
    nested keys the projection strips for a report declaring less, so every earlier attestation
    keeps verifying; schemas/report.v7.schema.json); the Markdown report and the compliance
    report render one sentence from provael.calibration.describe_calibration (tuned / bound /
    invalid) instead of each stating the split in its own words; the CLI table gains an eval FPR
    column. Old artifacts without the new keys load as what they are, two-way fits with no
    binding. No measured number moves, and no real-policy calibration validity is claimed: the
    roadmap's condition — a fresh supported real-policy calibration run — is unchanged, and every
    published number is still measured under the uncalibrated default box.

  • The brand assets under docs/assets/ (README wordmark, org picture, docs logo and favicon) now
    derive from the website's Proof-Path mark through scripts/gen_brand_assets.py; the teal
    checkmark this repository had carried since before the mark existed is retired everywhere, and
    the README wordmark gains a dark-scheme variant. The retired social_preview.png (the July
    "100 %" card) is deleted; the repository's social preview is the website's og.png.

  • The ML-BOM names the model it describes, and carries a serial number. Why: metadata.component
    is the component a CycloneDX BOM describes, and it named provael, the tool, so a consumer read
    the BOM as describing the red-team harness rather than the policy it measured; and the BOM had
    no serialNumber, which CycloneDX says every BOM should carry. The policy under test (its model
    card included) is now metadata.component, provael moves to metadata.tools.components, and the
    model is no longer repeated in top-level components. The serial is a UUIDv5 of the report's
    canonical JSON rather than a random UUID, so the same report still yields a byte-identical BOM.
    Every emitted BOM is now validated against the official CycloneDX 1.6.2 JSON schema, vendored in
    tests/fixtures/cyclonedx-1.6. Consumers that read metadata.component.name == "provael",
    or that looked for the model under components, must read metadata.tools and
    metadata.component instead.
    No measured number moves.

  • Attestations bind report.json's bytes, and the signature covers a small fixed record
    (provael-attestation/v2).
    Why: a v1 subject was the SHA-256 of the report's model
    projection, because the statement builder was handed a parsed report, so deleting a field whose
    stored value equals its default (loading restores it) left the digest intact; and a signed
    statement is 38 to 117 KB, while AWS KMS signs pure Ed25519 only over a raw message of at most
    4,096 bytes, which ruled out a KMS-held key for the operated tier. What changed: the subject is
    the SHA-256 of the file as written (subject.binding: report-json-bytes/v1, the value
    sha256sum report.json prints), so attest --verify --report is a byte-for-byte check; the
    Ed25519 signature covers a 306-byte signed record naming the statement's SHA-256 and the
    subject's, a 363-byte message under the DSSE encoding (attest.signed_record,
    record_signing_message). ReportArtifact carries the bytes read from disk to every statement
    builder (attest, certify, submit, the assurance and insurer views), submit publishes the
    submitter's report.json byte for byte instead of a re-serialised copy, the hosted server binds
    the request body as received (post it with curl --data-binary, not --data), report.json
    is written as LF bytes on every OS, and .gitattributes stops a Windows checkout rewriting
    byte-bound files. The verifier now also rejects a bundle whose payloadType is not an
    attestation, or whose statement format it does not know: before, a signature by the same key
    over any payload type verified as an attestation. The certify dossier no longer calls the bundle
    a PEP 740 attestation, which it never was. The committed insurer sample is regenerated as v2.
    Compatibility: v1 bundles still verify, by the v1 rule (frozen fixtures in
    tests/fixtures/attestation_v1 pin it), and the verdict names the rule applied; 0.44.0 and
    earlier cannot verify a v2 bundle, so upgrade verifiers before issuers. The execution and
    evidence manifests keep the projection digest, which is no longer equal to the attestation
    subject. No measured number moves.

  • Meta-World is declared scaffolding: the published counts move from 5 runnable / 2 scaffolding
    suites to 4 / 3.
    Why: no Meta-World benchmark has ever been run (nothing under results/), and
    its CLI path raises at reset, because the adapter refuses LIBERO's default keep-out box (it lies
    behind the Sawyer arm), there is no zone option, and no Meta-World calibration is committed. That
    is the project's own definition of scaffolding, and watch/registry.json still counted it as
    runnable. SCAFFOLDING_SUITES now carries it with that reason (the CLI-gating note is folded in,
    so SUITE_GATING_NOTES is empty); list-suites renders it as scaffolding, doctor now labels
    every scaffolded suite as scaffolding rather than "importable", and suite_is_ready("metaworld")
    is False whatever is installed. The docs and examples that described its status now call it
    scaffolding (quickstart, attacks, FAQ, index, roadmap, and the suites and cross-suite examples);
    ActionLayoutError's hint no longer suggests --suite metaworld, and examples/suites/README.md
    no longer calls its wiring "validated on the gated, real path", which the suite registry's own
    comment ("never been introspected against an installed package") contradicted.
    watch/registry.json and the generated inventory line in docs/quickstart.md are regenerated;
    the website picks the counts up at its next re-pin. That inventory line's three groups are now
    disjoint (3 CPU fixtures, 1 gated real simulator, 3 scaffolding): two suites are both gated and
    scaffolding, and counting them in both made the line sum to 9 of 7 (8 of 7 before this change).
    scripts/gen_doc_counts.py now refuses to render a line whose groups do not sum to the total.

Fixed

  • docs/attestation.md still said the instruction family "redirects the policy 100% (10/10)".
    Why: that is the single-task figure the ten-task suite superseded in August, stated in the
    framing E-2026-12 corrected, on the page that tells an auditor what a bundle proves. It now gives
    roleplay 42/50 against a 1/50 benign floor on 0.41.2 (44/50 against 2/50 on 0.32.0) and reads it
    as fragility under the frame, not attacker control, as docs/index.md does. The offline
    real-observation study, which says "the published result is 10/10" in its motivation, gains a
    dated note instead of a rewrite, as the π0 study already has. E-2026-17 records that ANSI/A3
    R15.06-2025 is carried at year precision. No measured number moves.
  • The signed regulatory clock dated ISO 10218 to 1 April 2025, a day no source gives (E-2026-17).
    Why: ISO's catalogue gives the publication date of both ISO 10218-1:2025 and -2:2025 as 2025-02,
    a month with no day, and the clock sits inside every signed statement, so every attestation from
    0.7.0 to 0.44.0 vouched for the unsourced day; the hosted conformity mapping typed the same day
    again. What changed: the entry reads applies_from: 2025-02; every clock entry carries a
    date_precision (day, month or year) and the model refuses a date written more finely than it;
    the entry's note no longer calls a published standard "in force"; the hosted conformity rows read
    their dates (and applies_from_precision) from the clock instead of repeating them; the ruleset
    moves to provael-attest-ruleset/6, because manifests stamp it as their
    regulatory_clock_version and two clocks must not share one; the committed evidence manifests
    that stamp it (results/smolvla_libero_object, the 14 September delivery pack) and the insurer
    sample are regenerated, and only that string and the clock move in them. No measured number moves.
  • list-suites called Meta-World ready on lerobot alone, and "horizon 280" had no stated
    source.
    Why: Meta-World is a separate lerobot extra that provael[lerobot] does not bring in,
    so a box with the extra was told the suite was ready and failed inside LeRobot's env factory;
    both readiness checks now require the metaworld module too, and the module docstring agrees
    with its own install hint. suites.libero.LIBERO_HORIZON records OpenVLA's per-suite evaluation
    budgets (220 / 280 / 300 / 520, each just above the suite's longest training demonstration),
    which every committed LIBERO run already used; the eai04-redirect recipe and the local sweep
    read it, the sweep no longer runs libero_spatial at object's 280 by default, and
    tests/test_libero_horizon.py holds committed runs to it. No measured number moves.
  • Four published docs claims the repository no longer supports. Why: each was found stale in
    the 26 September re-verification of AUDIT_BACKLOG.md and ISSUE_REGISTER.md. docs/faq.md
    and the instruction-transfer finding quoted Predictive Red Teaming as "MAE < 0.19" without the
    caveat docs/sim-predicts-real.md already carries (the result is on diffusion policies, not
    VLAs); docs/index.md listed GR00T and OpenVLA as real policies and Meta-World as a real
    simulator (both policies are unrun scaffolding, and Meta-World is not runnable end to end);
    docs/standards/listings.md said "nothing submitted yet" after sixteen listing PRs, and now
    carries the ledger its own checklist asks for; pyproject.toml called lerobot 0.6.0 the newest
    release (0.6.1 has been on PyPI since 3 August). The glossary now links the website's. No
    measured number moves.
  • Three module docstrings that had gone stale, and E-2026-15 in the errata log. Why: each said
    something the code no longer does. combine.py said provael attack cannot resume (the ledger
    has been wired in as --resume since 30 August, #157); attacks/controls.py said --attacks control resolves to "both arms" (it resolves to four); scripts/gen_brand_assets.py called the
    website upstream "for every public number" (it is upstream for the brand only). docs/errata.md
    records E-2026-15, the website's correction of two publication days its own regulatory clock does
    not verify; no measured number moves.
  • Four docs pages and suites/libero.py's module docstring said the contact column and the
    three-way calibration split shipped "since 0.45"
    ; there is no 0.45. Both are on main since
    21 September 2026 (#278, #279) and in no release yet, and all six lines now say so.
    tests/test_counted_claims.py fails on any tracked doc or src/ docstring that names a release
    after provael.__version__ — the mirror of the checks that catch prose falling a release behind.
  • Three ATLAS mappings named something that is not an ATLAS tactic. Why: checked on
    26 September 2026 against MITRE's data release 2026.09. EAI01 said "ML Attack Staging", a retired
    name (AML.TA0001 is "AI Attack Adaptation" since 2026.08), and ATLAS files neither prompt
    injection nor jailbreak under that tactic. It now reads Execution → LLM Prompt Injection and
    Defense Evasion → LLM Jailbreak. EAI02 said "Evasion" for "Defense Evasion". The Top 10's own
    crosswalk table already had both right. EAI03 put "ML Supply Chain Compromise", a technique
    under its retired name, in the tactic slot: it is AI Supply Chain Compromise, under Initial
    Access. The catalog, results/crosswalk/crosswalk.atlas.json, every SARIF rule's
    atlasTechniques and docs/standards/atlas-case-study.md move together. tests/test_eai.py
    now fails on a mapping whose tactic is not one of ATLAS's sixteen. The 8 August submission to
    MITRE is left exactly as sent. No measured number moves.

Verify what you downloaded

Every asset carries a Sigstore bundle (<asset>.sigstore.json) signed keyless by this
repository's release workflow, and a SLSA build-provenance attestation in GitHub's store.
Either route names the workflow, the tag and the commit that built the file:

# Sigstore bundle beside the asset (cosign >= 2.2)
cosign verify-blob --bundle provael-X.Y.Z-py3-none-any.whl.sigstore.json \
  --certificate-identity-regexp '^https://github.com/provael/provael/\.github/workflows/release\.yml@refs/tags/v' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  provael-X.Y.Z-py3-none-any.whl

# GitHub build-provenance attestation
gh attestation verify provael-X.Y.Z-py3-none-any.whl --repo provael/provael

SHA256SUMS is signed the same way; sha256sum -c SHA256SUMS then checks the rest.