Skip to content

SynthWorld 0.12.0

Choose a tag to compare

@github-actions github-actions released this 04 Aug 15:24
· 195 commits to main since this release
v0.12.0
47c1aa7

Changed

  • Breaking (ambiguity answer key): same_name_and_date_of_birth is now
    insufficient, not separate (#77). The pair is two people in canonical truth, but
    the public evidence — matching name and birth date, nothing distinguishing them —
    cannot justify concluding it. The pack's first consumer abstained on exactly this pair
    and independently gave the same reason. Membership truth is unchanged; the public and
    memberships artifacts are byte-identical, and only the dispositions artifact was
    re-cut (new digest in GOLDEN_REVIEW.md). A resolver scored against the old key that
    answered separate here was being rewarded for clairvoyance; one that abstains is now
    scored correctly.

  • EVALUATION_SCHEMA_VERSION is 0.2.0. TaskMetric gains optional family and
    support_meaning, so every task's report carries two more keys - extraction,
    entity resolution, relationship inference and risk included, even though none of
    their metrics changed meaning. The wire shape is what moved, so the schema knob is
    what moves; no per-task scoring version changes, because a scoring version here means
    the metric definitions and those are untouched. A stored 0.1.0 report does not
    load under the new model — the report's schema_version is a single-value literal —
    so read archived reports with the library version that wrote them.

  • Agentic metrics are grouped into five families and every denominator says what it
    counts, so a report can be read by family and each ratio re-derived rather than
    trusted. No metric value moves. The split carrying the most information is
    observability against the rest: an agent can decide well and record badly, or the
    reverse, and those need different fixes. Measured on the reference trace, wrecking
    the recording drops observability to 0.25 while identity resolution, authorization
    and delegation stay at 1.0; wrecking the decisions drops authorization to 0.40 while
    observability stays at 1.0.

  • AGENTIC_BENCHMARK.md gains a per-metric glossary: what each measures, what 0.0 and
    1.0 mean, and its denominator. Sixteen of the twenty metrics had no mention in any
    top-level document, so the only way to learn what they measured was to read the
    scorer or diff scores between policies. (They were cited in
    agent-authority-contract/control-catalogue.yaml and its design-intent notes, which
    a first version of this entry overlooked while claiming a repository-wide count.)

What's Changed

Full Changelog: v0.11.0...v0.12.0