3.0.0 - divergence and coverage, letter grades, vector search

This tag versions the suite and its methodology, not the standings. The board is regenerated several times a day, so the results table in this tree is a snapshot of 12 August 2026 and was already superseded within the hour. For live figures read /data/latest.json, /data/runs.json or a target's badge endpoint.

Pin to this tag when citing a finding, as CONTRIBUTING asks, rather than to main.

---

Read this first if you consume the JSON. The data endpoints go from schema 2
to schema 4 in one step. Schema 3 was never published on its own, so everything
on 2 crosses both steps at once. Schema 3 breaks in four ways:

• movement.state values changed from up/down to improved/regressed.
  This is the one that fails silently: the old names still parse, and now
  mean the opposite direction.
• movement.delta keeps its shape but is computed from divergence rather than
  correctness, so the sign of a delta means the opposite of what it did.
• A tier no longer carries pct and value. It carries divergence,
  coverage and correctness, each with a pct and a value of its own.
• The whole-suite correctness percentage is correctness, not total. total
  also names the raw test count inside counts, so the same word meant a count
  in one place and a percentage in another.

Schema 4 is additive on top: each target carries its grade and the full criteria
in metrics.grade, every envelope gains baseline.observation (how much of the
suite the live-AWS row stands on, which passes reported, and what is carried),
latest.json and runs.json gain divergence, project, configuration and
isVariant, results/summary.json gains regionFailures, and /data/index.json
lists the split registry.

A score is two figures, never one

Divergence is the share of the whole suite a target answers differently from real
DynamoDB. Coverage is the share it implements at all. They are reported apart and
never summed, because a declined operation is discoverable in minutes and a wrong
one in production. No target was re-run for the change and no pass, fail or skip
moved: what changed is how the same counts are expressed.

Everything else that was a percentage followed the headline down - tier figures,
the per-region drilldown, the per-operation table, and the colour bands, which
inverted with them. A target's history is two plots rather than one, because
divergence falls when a target stops attempting something it used to get wrong,
so a divergence line alone can render a withdrawal as an improvement.

Every target wears a letter

Divergence sets it - A under 5%, B under 15%, C under 25%, D under 35%, F beyond

• and coverage can only lower it, never raise it: a third of whatever a target
  leaves unimplemented is added to its divergence before the bands are read. A+ is
  exactly zero divergence at full coverage. A row says when coverage is holding its
  letter down, so a capped row is not mistaken for one with room above it. Real
  DynamoDB reads baseline rather than a letter, because grading the yardstick
  against itself would seat it in a band an engine had to earn its way into.

These are grading criteria version 1, dated in the
methodology, which carries the
derivation. Where a threshold sits is a hand-picked input to a published letter,
and moving one regrades targets whose results never changed, so any change to a
band, the coverage weight or the A+ gate bumps the version.

The suite grows from 998 tests to 1054

Vector search (#125). 42 tests over the deterministic surface DynamoDB
shipped in August 2026: the index lifecycle on both creation paths, request
validation on each plane, rejection wording, write-path validation, search on a
fixture where the nearest neighbour is unambiguous, the two new capacity shapes,
and PartiQL's inability to reach a vector index. Every pinned value was
characterised against real DynamoDB in eu-west-2 before it was asserted. Two
findings worth naming: searching during a backfill is an error, which settles
which side of a contradiction in AWS's own documentation is right - three
developer-guide pages say the call fails, the tutorial page says results can be
incomplete - and an overwrite leaving the stored vector unchanged reports no
vector write capacity at all, because index replication is delta-based. Both
sides are captured in captures/2026-08-12-vector-backfill-docs.json. No
emulator implements the family yet, so every target skips it and every coverage
figure drops with this release while divergence is untouched. Sending the new
operations needs @aws-sdk/client-dynamodb 3.1103.0 or later.

Index write costs (#124). 14 tests. The suite's only per-index capacity
assertion was on a Query, so the write side - the half you get billed extra for -
went unmeasured. A sub-1KB write costs one unit for the table and one for each
index it lands in, and LSI units fold into the total exactly as GSI units do.
Moving an item to a new GSI key costs two on that index, a delete and an insert;
touching a projected attribute costs one; touching a non-projected attribute
costs nothing, and the response carries no arm for that index rather than a zero.
An overwrite that leaves the item unchanged reports no index cost whatsoever.

The index exclusions create no indexed table (#116). !gsi and !lsi used to
select the right tests and then build the tables anyway, so an engine with no
secondary-index support died in setup whatever it had asked for. Shared tables
are created on demand from what the running file declared, the composite table
split into indexed and plain variants, and three guards keep the declarations and
the tags honest.

Corrections

• The suite counts its own tests. "The whole suite" had meant whichever
  target ran the most tests, so one of the measured things was setting the
  denominator every figure divided by. registry/suite-manifest.json lists every
  test by file and full name, generated from the suite rather than inferred from
  a run, and CI fails if it drifts. Publishing now refuses a row whose test
  population disagrees with it, which catches a results file carried across a
  rename that keeps its old total while naming tests that no longer exist.
• Every figure on a row comes from one region. A headline came from the
  target's best-matching region while its tier split and raw counts stayed on the
  baseline region's basis. Correctness never had to reconcile, because each tier
  had its own denominator; divergence is additive, so it does.
• The per-region overlay is matched to the run it describes, rather than
  keyed on the date real AWS was last swept, which had put a later run's figures
  on an earlier run's page.
• A methodology claim was wrong. The page said that measuring divergence over
  the whole suite stops an engine implementing a sliver from posting a perfect
  score. It doesn't: zero fails is 0.0% divergence at any coverage. What does hold
  is the identity the page now states - a test going from failing to skipped
  leaves both numerators together over the same denominator, so withdrawal costs
  exactly as much coverage as it gains divergence, and is disclosed rather than
  silent.
• The baseline row is measured rather than pinned once real AWS has been
  observed across the whole suite. It runs in three passes and only the main one
  reached the published artefact, so the row had claimed a full suite on a run
  that recorded less. The passes are merged before scoring, each with its own
  capture date, and the row stays pinned and says so until all three report.
• The Atom feed carries no letter on any run measured before criteria version 1
  took effect. It had been rewriting each entry's summary with a grade the run
  never had, while leaving <updated> alone so no subscriber re-notified.
• Badges publish the grade under a parity label, having still been
  publishing the correctness percentage the board retired under a conformance
  label. The endpoint URL is unchanged; a badge whose target has no results this
  run reads no data rather than disappearing, because the URL sits in other
  people's READMEs.
• Two more regional splits are admitted, both BatchGetItem with an empty
  RequestItems map. The nesting-depth row was rewritten from a full 32-region
  capture, having been written from four.
• A build of an engine nests under it rather than taking a row beside it, and
  maintainedByAuthor is keyed on the project, so it now reads true for the
  WebAssembly build. That build runs in CI like every other target, where its row
  had been refreshed by hand.
• The board leads with the highest-graded engine, since the baseline moved into a
  panel above the standings. Where that is the board author's own engine, the
  conflict-of-interest disclosure sits on the card.
• The Region column is gone. It named the cohort a target matched at its best
  rate, which read as breadth. The count sits beside the figure instead, and the
  cohort listing stays on the target page.
• Each target lists how it is actually distributed, with the project's own page
  for each. These are the only claims on the board the suite does not measure, so
  each carries the link that backs it.