Repository navigation
Releases
v0.7.1
Added
- Edge vector EV-114 (113 → 114): a
codemapping.
Changed (PRML v0.1 erratum 2026-11-15, third part)
codeis accepted as an optional mapping, as spec §2.2 has listed it
since the first draft; all four implementations rejected it as an unknown key.
Any other type is rejected.- §2.2 lists
compute_envelope, which the schemas and implementations
already accepted. - JSON Schemas widened to the text:
modelallows keys besidesid,hash
anduri; the v0.2 schema listsmetric_argsandcode; its v0.2 RFC
proposal fields are marked as not adopted. The schema page says the text
decides where the two differ.
v0.7.0
Added
- Conformance vectors for the second part of the erratum: edge suite
EV-109..EV-113 (108 → 113) and reject suite RJ-041..RJ-060 (40 → 60; 6 of the
new 20 are YAML-only, so 22 of the 60). No existing vector changed.
Changed (PRML v0.1 erratum 2026-11-15, second part)
- Mapping keys must be strings. A YAML key read as a number, boolean, null,
sequence or mapping (1:,true:,~:,? [a, b]) is rejected with exit 2;
quote it for a string. Python and JavaScript used to write such keys differently. - U+FFFE, U+FFFF and U+10FFFF are rejected in any string or key, as the other
non-portable characters are. The other Unicode noncharacters stay allowed. - Field types are checked in all four implementations, as the published schema
states them:comparator: null,metric: 123,dataset: "x"and similar
manifests no longer lock or verify.notes: nullandprior_hash: nullare
rejected by the Python reference too.modelis checked for its type only,
as before. The schemas'seedno longer has a negative branch, matching §2.1. lockrequirescreated_atasYYYY-MM-DDTHH:MM:SSZ.hashandverify
still read any RFC 3339 spelling, so manifests locked earlier keep their hash.- A nested
threshold(insidemetric_args) renders by its spelling in
impl/js,impl/goandimpl/rust, as the Python reference and the grammar
already did (grammar G4). Their hashes change for such manifests only. - Tests: the shared erratum fixture grows to 67 cases (40 rejections), with a
locktest forcreated_at.
v0.6.0
Added
- Conformance vectors for the erratum of 2026-11-15: edge suite EV-093..EV-108
(92 → 108) and reject suite RJ-021..RJ-040 (20 → 40; the new 20 are 12 YAML and
8 JSON, so 16 of the 40 are YAML-only), one input per rule. A reject vector'sexpectmay now list several accepted
substrings. No existing vector changed.
Fixed
- Mapping keys in
impl/js,impl/goandimpl/rustnow follow the §3.6 C5
plain-scalar rule, as the grammar and the Python reference already did: a key that
is not a plain scalar renders single-quoted. Manifests whose free-form keys
(metric_args,model,compute_envelope) need quoting now produce the same
canonical bytes in all four implementations; their JS, Go and Rust hashes change.
No published vector changes. impl/rustnumber parsing uses correctly rounded float parsing
(serde_jsonfeaturefloat_roundtrip). Some 16 to 17 significant-digit values
were previously read as the neighbouring float64 and canonicalized to different
digits than the other implementations. No published vector changes.impl/gonumbers render by value, as in the other implementations: a decimal
or exponent spelling is read as a float64 and rendered by §C4, so0.850,85e-2
and0.85give one hash, and an integer spelling stays exact. A number that does
not fit a finite float64 (for example a1e999threshold) is now rejected.
Manifests that spell a number with trailing zeros or an exponent get a new Go
hash. No published vector changes.- Stricter JSON input in
impl/goandimpl/js: invalid UTF-8 and, in Go,
content after the JSON value are rejected, as in the Python and Rust
implementations; a lone surrogate escape is rejected, as in Rust. impl/jsreads numbers from their spelling, with its own strict JSON reader
(same behaviour on every Node.js version): integers beyond 2^53-1 stay exact in
JSON and YAML,seed: 42.0is rejected, and a float written1.0renders1.0
(§C6), as in the other implementations. Manifests with integer-valued floats
outsidethresholdget a new falsify-js hash.impl/jskey order follows Unicode code points (§C2) for keys above U+FFFF.- Duplicate-key check in
impl/js,impl/goandimpl/rustno longer treats a
string inside an array as a key name:["a", "b", "b"]is accepted. impl/rustkeeps integers beyond the 64-bit range exact (serde_json
featurearbitrary_precision) and rejects a sequence nested in a sequence
instead of panicking.
Changed (PRML v0.1 erratum 2026-11-15, see spec/PRML-v0.1.md Change Log)
- YAML input: a plain scalar is a number only when spelled as a JSON number
(1e3is now the float 1000.0 in the Python reference too); other numeric-looking
plain scalars (010,0x1F,1_000,1:30,.5,+1) and unquoted timestamps
are rejected with exit 2. Quote them for a string.<<and=are plain strings. - Rejected in all four implementations: non-finite numbers in any field, a
seedoutside 0 to 2^64-1, a sequence directly inside a sequence, nesting deeper
than 100 levels, and lone surrogates (U+D800 to U+DFFF). Keys named
constructor,prototypeand__proto__are accepted everywhere. - Canonical form: an empty mapping renders
{}and an empty sequence[]
(grammar C8;impl/js,impl/go,impl/rustand the registry change, Python
already did this). When two shortest digit strings tie, the one with the even
last digit is used (grammar C4;impl/rustchanges). - Python reference (
falsify_prml.py) follows the grammar where it had
departed: a string beginning---or...renders plain when §C5 says so, a key
longer than 128 characters or an empty key renders as a simple key, and a YAML
anchor or alias is written out instead of emitted as&id001. YAML tags outside
the core types (!!binary,!!set,!!omap,!!pairs,!!timestamp) are
rejected. Manifests containing these inputs get a new Python hash. - Legacy
falsifyCLI (falsify.py) rejects a repeated key in a spec file. - Tests:
tests/test_erratum_2026_11_15.pychecks one case per erratum rule
(tests/fixtures/erratum-2026-11-15-cases.json) against the Python reference and,
where available, the JavaScript, Go and Rust CLIs; the multi-language workflow
runs it for each implementation. The JavaScripttest-vectorsrunner now reads
vector files with the strict reader, so inputs keep their exact spelling.
v0.5.3
Changed
- Repository links in package metadata, README, CITATION and docs point to the
falsify-devGitHub organisation. No behaviour change. - Conformance test fixtures:
producer.idis nowfalsify.dev. Every
v0.1, v0.2, edge, reject and candidate vector (and the generators and tests
that build them) used a former organisation domain as the producer id. The
expected canonical bytes and SHA-256 digests were regenerated with the
reference generators and checked across all four implementations (13 + 8
positive, 92 edge, 20 reject). Canonicalization rules and vector counts are
unchanged. New TV-001 digest:47f9956b3b9c495b3bf03b67398f54d0aa2133a2faf9b0522a9efa90bb109466.
v0.5.2
Fixed
- The wheel ships
falsify_linkage.py. It was missing frompy-modules, so
in 0.5.1falsify-engine linkage ...failed withModuleNotFoundErrorand
falsify linkagewas not a subcommand, although the CHANGELOG (2026-08-13)
andspec/linkage/prml-linkage-0.mdname the Python reference module. falsify linkage start|finish|verifynow works from thefalsify
command: arguments are forwarded unchanged to the engine CLI, so both
entry points behave identically.
Added
tests/test_packaging.py: every repo-root module imported by a shipped
module must be inpy-modules;falsify linkageround trip (start, finish,
verify) through thefalsifycommand.- Release workflow installs the built wheel in a fresh venv outside the
source tree and runsfalsify --version,falsify linkage --helpand
falsify-engine linkage verify --helpbefore publishing.
v0.5.1
Added
--jsononfalsify verify,lockandhash. One JSON object on stdout
with stable keys (status,exit_code,sha256,recorded,metric,
comparator,threshold,observed, ...); exit codes are unchanged.
statusseparates the two cases that share exit 11:NO_LOCK(no sidecar or
--expected-hash) andDATASET_MISMATCH. For CI steps and coding agents
that should not parse text. Tests:tests/test_prml_json.py.
Every error path (unreadable or invalid manifest) also returns one object,
{"status": "ERROR", "exit_code": 2, "error": ...}, on all three commands;
hash --jsoncarriesstatusandexit_codelike the others.
v0.5.0
Added
-
falsify anchor <spec> [--sealed] [--registry URL] [--handle H]. Posts the
manifest to a PRML registry (default registry.falsify.dev), refuses the answer
unless the registry's hash equals the local canonical hash, and saves
<stem>.prml.receipt.json,<stem>.prml.tsr(RFC 3161) and
<stem>.prml.rekor.json(Rekor v2) next to the manifest.--sealedpublishes
only the hash and timestamps. Standard library only. -
falsify receipt verify <spec> [--fetch] [--trusted-root FILE] [--json].
Offline receipt check without OpenSSL: the RFC 3161 token's messageImprint must
equal the manifest hash and its signature must chain to Sigstore's TSA root;
the Rekor entry's digest must equal the manifest hash, its RFC 9162 inclusion
proof must reach the checkpoint root, and the checkpoint must be signed by the
log key. Trust anchors are pinned from Sigstore's trusted_root.json (sha256
6494e21e…, 28 Oct 2026);--trusted-rootoverrides them. Exit 0 VERIFIED,
3 INVALID, 11 PARTIAL (log key unknown), 2 bad input. Needs the new optional
extrapip install "falsify[receipt]"(rfc3161-client, cryptography).
Tests:tests/test_receipt.py(offline fixtures from a real registry record,
mock registry foranchor). -
falsify attest <spec> --as test-result --observed <v> [--subject NAME=sha256:HEX] [--url URL].
Emits an in-toto Test Result v0.1 Statement for one run:configurationis the
PRML manifest (digest = PRML hash, media typeapplication/vnd.prml+yaml),
resultis PASSED or FAILED from the locked predicate,subjectis the
tested artifact (default: the dataset). It refuses to attest when the manifest
no longer matches its lock (sidecar or--expected-hash; exit 3) and exits 10
on FAILED so CI can gate. In GitHub Actionsurldefaults to the run URL.
Hosts that already consume in-toto test results can require "ran against this
locked bar" without learning a new predicate type. Tests:
tests/test_attest_test_result.py.
Changed
falsify initwrites a manifest that can be locked as written. It now
generates a real UUIDv7claim_id(RFC 9562;uuid.uuid7is 3.14+ only) and
the current UTCcreated_at, and--dataset FILEsetsdataset.hashto the
file's SHA-256 (dataset.iddefaults to the file name). New flags--metric,
--comparator,--threshold,--seed,--producer,--dataset-id. A
missing output directory is a clean error (exit 2) instead of a traceback; a
.jsonname writes JSON. Placeholders that remain are listed. Tests:
tests/test_prml_init.py.- Docs:
docs/CASE_STUDIES.md,docs/EXAMPLES.mdanddocs/DEMO_SCRIPT.mdused
the old command name for the workflow engine (falsify init --template,
falsify run,falsify verdict, ...); they now sayfalsify-engine.
v0.4.1
Added
spec/COMPATIBILITY.md— compatibility and deprecation policy, adopted
2026-09-13 and linked from the specification's Status section: what a
specification version, a package version and a registry record promise; how
anything is deprecated (two minor releases or six months, stderr warning,
migration note); vectors are added and withdrawn, never changed.- v0.2 forward-promises dispositioned (Errata 2026-09-13, editorial): metric
registry withdrawn; signature sidecar re-scoped tospec/v0.3-backlog/06-signature-sidecar.md
with an Ed25519 test vector; noproducer.tierfield, ever.
Fixed
falsify verify --observed infno longer passes.float()accepts
inf/nan, and an infinite observation satisfied every>=bar (exit 0).
The Python CLI and the Go reference CLI now reject non-finite observations
with exit 2, as the JS and Rust CLIs already did. Regression test:
tests/test_prml_verify_observed.py. Reported by the 28 Oct 2026 code audit.
Changed
- README quick start: states the Python 3.11+ requirement (with
pipx/
uvxfor older system Pythons) and adds the missinggit clonestep before
cd examples/first-manifest.
v0.4.0
Canonical output changes for some inputs (v0.2 integral-float thresholds), hence a
minor bump, not a patch. The reference canonicalizer now implements the §3.6 grammar's
C6 rule for v0.2 and ships with the grammar, the 92-vector edge suite and the
implementer's guide.
Fixed
examples/template.yamlno longer reads as a PRML manifest. It is the
workflow engine schema and always was, but it sat unlabelled among the PRML
examples and the header told readers to runfalsify lockon it — which exits 2
with "missing required field: version / claim_id / created_at". The header now
names the right tool and the right flow (falsify-engine init <name>, which takes
a claim NAME, not a file path), and points PRML users atfalsify initor
examples/first-manifest/accuracy.prml.yaml.README.mdcalledfalsify.pythe Python reference implementation. It is not;
falsify_prml.pyis.falsify.pyis the workflow engine, a different tool with a
different schema, installed asfalsify-engine. Two places corrected.
Added
examples/README.md— routes a reader to the right tool in one table. Every
command in it was executed before the file was committed.README.mdreference-implementation section now states which implementations are
distributed (Python on PyPI, JavaScript on npm) and which exist in-repo only
(Go, Rust), with each one's command set. Go and Rust implement the verification core
and deliberately omit authoring commands; there is no Go module and no crates.io
crate, and none is planned for the 0.x line. Previously a reader had to guess.pyproject.tomldescription now says the package also shipsfalsify-engine.- §3.6 Formal Grammar (
spec/grammar/): an ABNF for the canonical byte
sequence plus constraints C1–C7, normative. Verified in both directions by
spec/grammar/check_grammar.py, an emitter and recogniser written from the
grammar alone (no PyYAML): 21/21 conformance vectors reproduced byte-for-byte,
16 malformed canonical texts rejected.tests/test_grammar.pyruns the same
checks in CI. No canonical byte or digest changes. spec/grammar/candidate-vectors-2026-09-13.json: 83 inputs at the edges of
the plain-scalar predicate and the float rule, with expected bytes and hash
and each reference implementation's agreement on 2026-09-13 — kept as the
dated pre-correction snapshot; 82 of the 83 were promoted to the edge suite the
same day (see below).
Changed
- §3.5 no longer calls the reference canonicalizer "normative for this
rendering"; constraint C4 of the grammar is normative and the canonicalizer
conforms to it. Recorded as a dated erratum in the spec. multi-lang-conformance.ymlsummary notice said "14 reject-vectors"; the
suite has 20 (16 for the JSON-only implementations) since 0.3.13.
Fixed (cross-language, same day as the grammar)
- JavaScript, Go and Rust reference implementations corrected to the §3.6
grammar. The 83-input battery run through all four implementations gave 62
agreements and 21 divergences — none covered by a conformance vector, so the
daily CI had stayed green. The three hand-rolled plain-scalar predicates
(ported from one another) over-quoted?x:xyn1e50o17and
under-quoted-<<=1_0001:300b101; Rust rendered athreshold
of1e-05as0.00001. All three predicates are now transcriptions of README
§C5 (P1–P6 plus the YAML 1.1 resolver patterns verbatim); Rust's float
rendering implements §C4 from the exponent rather than from ryu's choice of
notation (Rust gains theregexcrate). The registry'scanonical.jswas
corrected in step; all six publicly listed registry records re-derive their
stored hash under the corrected canonicalizer. Result: Python 83/83, Go 83/83,
Rust 83/83, JavaScript 82/83 — 83/83 in all four after the v0.2 decision below. - Edge suite
spec/test-vectors/edge/edge-vectors.json(92 vectors,
EV-001..EV-092) added and wired intomulti-lang-conformance.ymlfor all four
implementations. Separate from the 21 normative vectors of Appendix B by
design (spec/test-vectors/edge/README.md).FACTS.envgains
VECTORS_EDGE=92.
Changed (specification, v0.2)
- v0.2
thresholdcanonicalizes by value (RFC "Post-freeze clarification
(2026-09-13)"): integral and below 2^53 → integer digits, otherwise the §3.6 C4
float.1300.0and1300are one manifest. The frozen RFC had never stated a
rule; the ROADMAP proposed the opposite of what the vectors did; JavaScript and
the registry could not observe the type distinction at all. Python, Go and Rust
now render1300.0as1300; no published digest changes (no vector spelled an
integral threshold as a float).falsify_prml.canonicalizeapplies the rule at
the root mapping only, like the v0.1 float rule. - Edge suite grows to 92 (EV-083..EV-092 pin the boundaries:
-0.0→0,
2^53-1,2^53→9007199254740992.0,1e16/10000000000000000→1.0e+16,
1300.5,1e-05).FACTS.envVECTORS_EDGE=92. All four implementations and
the registry pass 92/92.
v0.3.14
Packaging and presentation only. No change to canonicalization, verification
or exit codes; falsify verify behaves identically to 0.3.13.
Changed
- The README no longer describes the current product as a three-day
hackathon build. The README is this package's long description, so that
line was the first thing a reader of the PyPI page saw about what falsify is
today. The origin is not removed: the 0.1.0 entry below andSUBMISSION.md
both still record it, with dates. - Package keywords dropped
claudeandhackathonand now read
falsifiability, pre-registration, ai-evaluation, reproducibility, audit, CI.
Neither removed keyword described what the package does.
Fixed
falsify stats --htmlemitted a broken link. The generated report footer
pointed atgithub.com/<USER>/falsify-hackathon, a placeholder that never
resolved, in every HTML report a user produced. It now points at
github.com/studio-11-co/falsify.