Skip to content

v0.3.2 — topology benchmark: checked quickstart, generated card, reconciliation

Latest

Choose a tag to compare

@joe-carr-data joe-carr-data released this 09 Aug 19:29

v0.3.2 supersedes v0.3.1 and v0.3.0. Use this release. The 274-case
payload is byte-identical to v0.3.1; what is new is a checked quickstart,
a generated benchmark card, and the per-segment reconciliation every
population figure is now derived from. Earlier defects are listed under
Corrections.

Start here:

tar xzf windcheck-v0.3.2-topology-benchmark.tar.gz
cd windcheck-v0.3.2-topology-benchmark
./tools/quickstart.sh

That fetches one small published trace, scores it as a candidate and
shows it rejected because it self-intersects, rebuilds the reference
derivative from the shipped mask and shows it scored, then tells you
where to substitute your own. Both demonstrations are gated: if the
validator cannot be reached, the census runs over an empty surface, or
the reconstruction is not the packaged reference, the script fails
instead of printing a passing demo.

Measure a different method on the same problem

v0.1.0-beta and v0.2.0 published a repaired, topology-certified corpus:
274 censusable Herculaneum traces supplied with transverse-clean
reference derivatives — 184 transformed and 90 already clean — in the
format the pipeline reads.

This release makes that corpus usable as an evaluation set. For every
case it ships the input census witness — empty for the 90 already-clean
inputs — exactly which cells the reference transformation removed, and an
evaluator that scores any candidate derivative under the same rules.

If you have a method that repairs self-intersecting surfaces, this lets
you measure it the way ours was measured.

What is in the archive

file what it is
README.md the evaluation contract and every scope statement
index.json 274 packaged cases, 10 excluded with evidence
locators.json where to fetch every operand — 274/274 inputs, 274/274 references
traces/<segment>/ witness.json, witness.npz, masks.npz, fields.json, provenance.json
tools/ topology_eval.py, quickstart.sh and twelve supporting tools
CARD.md one page generated from the records below; refuses to build if it disagrees with them
reconciliation.json one row per segment; every population figure here is derived from it, not retyped
B7-acceptance.json, negative-controls.json, witness-binding.json, selfcheck.json the records described below

5,241,478 self-intersection contacts are shipped as witness data,
each naming the participating quads and triangle indices.

Inputs and reference derivatives are not re-hosted. They are already
published — upstream for the original traces, in this project's releases
for the derivatives and repaired bases — and locators.json gives each a
URL or bucket prefix, archive sha256, member path and fetch command. The
reference is also reconstructible from the input and the shipped mask:
stamp -1 at the removed cells and the complete arrays match.

The rules, up front

These are reference derivatives, not ground truth. Nothing here
establishes that the removed geometry was wrong papyrus, that the
retained geometry is right, or that a different cut would be worse. The
benchmark measures agreement with a topological invariant and the cost of
achieving it. It says nothing about the scroll.

The reference is one admissible answer, not the target. A candidate
that stays clean and unfragmented while retaining more area is better,
and the evaluator says so rather than scoring distance-to-reference.

No single scalar score, ever. A leaderboard number would invite
trading fragmentation against retention, and those are not exchangeable:
a hairline cut that splits a sheet in half still scores 99.99% on area.
The evaluator emits a vector and refuses to collapse it.

A candidate that is not clean is not scored on cost at all — otherwise
a method that did nothing would rank first. A census over zero triangles
is refused rather than reported clean.

What was checked before this went out

  • Acceptance test — 274/274. Scoring each reference derivative as if
    it were a candidate reproduces its published certificate: clean
    verdict, retained fraction, removed and added cells, component counts
    where recorded by the certificates (the 184 transformed cases), both
    R_main statistics, the 99.9%-area core gate, coordinate fidelity. A field present in the certificate but absent from the
    evaluator counts as a failure, not a skip. The census came from the
    official vc_tifxyz_selfcross over all 274 reference surfaces:
    274/274 clean under both diagonals, none vacuous, 456,475,808
    triangles.
  • Witness bound by identity — 274/274. Every input was recensused
    from scratch and its witness required to equal that census as a
    multiset of canonical contact identities, not merely to match on
    counts. All 5,241,478 contacts.
  • Controls — 7/7 negative rejected, 1/1 positive accepted. Counting
    them together as "8/8" misdescribes the suite, because one of the eight
    must be ACCEPTED. Seven negative mutations are rejected: a dirty
    candidate, an empty one, one with a moved coordinate, one with
    reinstated geometry, a different input of the same grid, a clean census
    report belonging to another surface, and an unbound report. The eighth
    is a positive regression control requiring that a relative candidate
    path census successfully. Each negative control must be caught for its
    own reason, so an evaluator that refused everything would fail rather
    than pass them.
  • Independent verification — 274/274, without importing the packer.

The acceptance test found two defects in the evaluator before release —
it was pricing quads the census drops for edge length, and counting
zero-area components — and both were fixed. That is what it is for.

What is new in v0.3.2

The 274 traces, their witnesses, masks and topology_eval.py are
byte-identical to v0.3.1. Everything below is additional.

  • tools/quickstart.sh — the contract in one command, and it is
    CHECKED rather than narrated. It previously would have treated any
    non-zero exit on the original trace as success, which is also what a
    dead validator, an unmounted path and a malformed report produce.
  • CARD.md — one page generated from the package's own records,
    which refuses to build rather than print a figure it has not
    established: a missing verification record, an acceptance record not
    covering the packaged population exactly, an evaluator/certificate
    disagreement, a partial result, or an exclusion without the
    confirmation the card claims.
  • reconciliation.json — one row per segment, from which every
    population figure is derived instead of retyped. It refuses when two
    source records disagree about a segment.
  • Archive construction is deterministic for the frozen inputs used
    here.
    Two builds from the same committed tools and frozen payload
    produced identical bytes. All 14 archived tools are byte-identical to
    the public tag target. The tag does not carry out/benchmark, so a
    clean checkout of it alone does not rebuild the archive.

Scope, stated plainly

  • 284 traces indexed, 274 packaged. 10 excluded as not censusable,
    below the 5,000-cell validity floor, each listed with its valid-cell
    count and confirmation that the census declined it.
  • The inputs are 171 published original traces and 103 certified
    displacement-repaired bases
    . The repaired bases are this project's
    artifacts, not upstream publications, and are named as such.
  • Six of the 274 reference derivatives are transverse-clean but fail
    the preregistered 99.9%-area core fragmentation gate.
    The worst
    retains 0.827046 with minimum core R_main 0.000
    (20251217234605-w2_20251217234605189); the other five retain
    0.946785-0.989331. The per-segment verdicts are in
    B7-acceptance.json. Named here so the aggregate does not hide them.
  • Area and connectivity are measured under certificate semantics. The
    official loader additionally invalidates z <= 0, so on traces with
    retained geometry below that plane this measurement covers a superset
    of what the official validator examines. Each result reports
    n_retained_cells_at_z_le_0.
  • The corpus is a dated snapshot of what was published at capture time,
    not a live mirror.
  • mask.tif must use 255 for retained cells: the official loader
    keeps a cell only where the mask is >= 255, and a mask written as
    {0,1} loads as an empty surface. This project shipped that defect in
    its earlier releases and corrected them; the check that would have
    caught it is now part of the evaluator.

Getting started

tar xzf windcheck-v0.3.2-topology-benchmark.tar.gz
cd windcheck-v0.3.2-topology-benchmark

./tools/quickstart.sh            # the whole contract, on one small trace

python tools/topology_eval.py \
  --input      <the input tifxyz> \
  --candidate  <your derivative> \
  --provenance traces/<segment>/provenance.json \
  --reference  <the reference derivative> \
  --validator  vc_tifxyz_selfcross

--provenance is what makes "same input" mean same bytes. --validator
is what makes the clean verdict authoritative — the evaluator runs the
census itself, and a census it did not run is never marked authoritative.

Questions, disagreements and bug reports are welcome on the issue tracker.


Corrections

v0.3.0's evaluator failed on relative paths

v0.3.1 replaces v0.3.0. If you downloaded v0.3.0, use this release
instead.

What was wrong

The v0.3.0 evaluator passed the candidate's path to
vc_tifxyz_selfcross exactly as given. That validator is normally run
through a container launcher whose working directory is not yours, so a
relative --candidate path resolved to nothing inside the container
and the census failed with:

Error: failed to load surface from <path>: Cannot open: <path>/meta.json

which reads as a broken surface rather than a broken invocation. Absolute
paths were unaffected.

What was NOT wrong

The benchmark data and scientific records are unchanged: witnesses,
masks, certificate fields, provenance, locators, B7 acceptance, witness
binding and self-check records remain byte-identical.
negative-controls.json changed from seven negative controls to seven
negative plus one positive regression control; all eight pass. Code changes are limited to
topology_eval.py, that regression control in benchmark_mutations.py,
and the new audit_archive.py.

How it was found, and how it stays found

By cold-running the published v0.3.0 package from a fresh extraction of
its own bytes — the first invocation failed. None of the published
results depended on the defect, which is exactly why nothing caught it:
the acceptance test supplies precomputed census reports and the
reproduction script passes absolute paths. The mutation suite now carries
a positive control requiring that scoring through a relative candidate
path still censuses, so it cannot regress silently.

Identity

v0.3.2 archive sha256 83919df3cc67d095d3d8dce7b21dbaa6cf5d967d0010edcdc7fea530eedede6b
v0.3.2 archive bytes 12,895,694
v0.3.2 tag target b9f5e1364bd624b9912db7743cc6d6ea66d6e090
v0.3.1 archive sha256 3b7da7d4c9e5c9d204f300e136fd4c8e021109fa5eb7bfc8f51fd9a3d89d18fc
superseded v0.3.0 archive a36c91d43c5c6e0153e73625676a963c2b02718d6327a4cfa8bfa04ec551dd59

The v0.3.1 tag is commit 0b3fbab, which contains the fix. v0.3.0
remains at 6b083bf and is not moved.


Correction - 10 August 2026. These notes originally said that one of
the 274 reference derivatives failed the fragmentation core gate. That
count applied only to the 99-trace expansion inventory. Across the
complete benchmark, six of 274 reference derivatives are
transverse-clean but fail the preregistered 99.9%-area core fragmentation
gate. The worst retains 0.827046 with minimum core R_main 0.000; the
other five retain 0.946785-0.989331.

No archive data or evaluator result changed. The shipped
B7-acceptance.json has always contained the evaluator and certificate
verdict for every segment, with agreement on all six failures. The error
was in the release-note scope, not the packaged measurements.