Skip to content

GitDuck Bench V1.1.0 — The Geese Have Rankings

Latest

Choose a tag to compare

@atlas-ocm atlas-ocm released this 04 Aug 19:15

V1.1.0 adds reproducible derived views over the immutable V1 result corpus.
No benchmark case, scorer contract, or historical attempt record was changed.

28 attempts · 0 ducks
Distance-to-Duck leaderboard
Outcome-awareness diagnostic
6 exact-text collision events across 5 shared digests

Finding the load-bearing key is not equivalent to providing exact
provenance. The awareness table preserves detection; the leaderboard ranks
precision.

Distance to Duck          = unofficial derived leaderboard
Outcome Awareness         = diagnostic projection, not a ranking
DUCK 1/1                  = only official benchmark metric
🔑                        = badge, never a sorting bonus
Δexact                    = defined only for eligible correct-action answers

Distance-to-Duck leaderboard
— Δexact is the multiset symmetric difference over all exact-set fields
of the answer contract (support keys + evidence refs + blocked set), defined
only when schema, authorization and action selection all PASS, so Δexact = 0
⇔ DUCK 1/1 — asserted per row by the verifier. It reads as an edit distance:
the number of single-element fixes to the exact support contract. Dense
ranks; chronology never breaks ties; byte-identical attempts share a rank
but stay separate attempts; raw-model and agent-product lanes never mix.

Closest attempt       Claude Sonnet 5 — Δ 1 (one extra related support key)
Best clean miss       nemotron-3-nano:30b — Δ 2 (one wrong key, zero fabricated refs)
Outcome-key citers    4/28; first: minimax-m2.7 (attempt 12)

The Outcome-awareness projection (second table of the leaderboard) keeps
detection visible without rewarding it: four attempts really did see the
load-bearing key; Sonnet used it almost correctly, two key-bearers caught it
in a net along with the whole pond.

Exact-byte convergence
— the collision ledger ranked as events, never as synthetic pairs:
6 collision events · 5 shared answer digests · 4 cross-provider · 1
intraprovider · 1 original same-infrastructure event that remains UNVERIFIED
FOREVER. Exact bytes do not prove copying, distillation, caching, aliasing,
or a shared physical backend.

Both tables are generated from the published records and regenerate
byte-identically:

node tools/verify-manifest.mjs
node tools/verify-results.mjs
node tools/verify-derived-tables.mjs

Manifest SHA-256:

9db6563b55d3d4c6dcb6316306a72073e23cc910d4669471a2091992c66a5278

Also in this release: the goose is drawn as a swan 🦢 — the goose glyph
U+1FABF is Unicode 15 and renders as a box on Windows 10 and other frozen
emoji fonts. The canonical award name is unchanged: GOOSE everywhere data
speaks; the swan is its well-supported stage mask.

GitDuck Bench V1.1.0 — The Geese Have Rankings. 🦆🦢📊