Skip to content

v0.5.1

Choose a tag to compare

@github-actions github-actions released this 05 Aug 05:47
· 9 commits to main since this release
v0.5.1
077ecb1

Added

  • 15-trust-repair pre-pilot scaffold — arms, runner, and oracle wiring
    for the trust-boundary-repair workload (decision of record for the second
    workload; full corpus and confirmatory runs remain gated).

Changed

  • 14-sig-triage: the ablation findings are now pre-registered and
    replicated.
    Six contrasts were frozen at main commit 96cc418 before any
    confirmatory cell ran; the battery executed 11 arms × 3 draws (5× on the one
    contrast that hit the registered boundary trigger), and five of six
    survive Holm–Bonferroni at α=0.05. Published figures move accordingly, and
    every movement is against the author's interest: decomposition depth from
    −9.2 to −5.2, retrieval payload from +6.4 to +3.0, and retrieval
    payload further from RESOLVED to UNRESOLVED under Amendment 2. The
    headline span widens to 59.6→81.6% because a registered S1 draw is now
    the lowest archived cell. New rendered page: eval/results/CONFIRMATORY.md.
  • 14-sig-triage: the confirmatory estimator is now issue-clustered.
    Amendment 2 replaces pooled McNemar as the primary test. Pooling concatenated
    repeated measurements of the same 250 issues and treated them as independent
    trials, which overstates significance; the primary is now a sign-flip
    permutation test over whole issues. Effect sizes are unchanged — only the
    uncertainty was wrong. Cost: contrast 2 (retrieval payload) drops to
    unresolved. Pooled McNemar remains in the output, labelled descriptive.
  • 14-sig-triage: "a bad harness scores below no harness at all" is
    retired.
    Across replication the fresh-session scheme (61.3%, 5 draws) and
    the bare model (61.9%, 4 draws) are statistically level. The claim is now
    that a bad harness ties the bare model while paying for an extra stage and
    250 tool calls — which is the stronger statement anyway.

Fixed

  • temperature: 0 is finally reachable from a single setting (#56) —
    greedy decode required setting temperature in both config.yaml and every
    agent's frontmatter, because the frontmatter parser defaulted to 0.7 (which
    shadowed config) while resolveAgent used 0 as its unset sentinel (which
    made an explicit agent-side 0 read as "unset"). Set-ness is now tracked
    through frontmatter and lifecycle YAML (TemperatureSet, mirroring
    EnabledSet), the built-in 0.7 lives in exactly one place
    (config.DefaultTemperature), and the documented priority lifecycle >
    frontmatter > config.yaml > built-in
    actually holds. The agent-config debug
    line logs temperature_source=agent|config so this class of shadowing is
    visible, and leather doctor gained a temperature row.
  • leather doctor no longer fabricates config problems (#31) — source
    attribution used a value != default heuristic, so a value explicitly set
    equal to the built-in default was labelled default, and the catch-all
    config/env/flag never said which layer won. config.Load now records true
    per-key provenance and doctor reports it: yaml, env, flag, or
    default. Also fixed en route: LEATHER_PERSIST_RUNS_DETAIL and
    LEATHER_PERSIST_RUNS_TOOL_CAP were only read at startup seeding, so a YAML
    value silently beat those two env vars, contradicting the documented order.
  • The documented config precedence was wrong, not the code (#33) —
    docs/modules/config.md claimed YAML overrides environment variables; the
    code has always applied env over YAML. The doc now states the real order —
    flags > env > YAML > built-in — plus the config-file resolution chain, and
    a test pins env-over-YAML for llm_endpoint, the field from the original
    wrong-endpoint incident.
  • Falling back past a project-local config.yaml is now loud (#30) —
    invoking leather bare in a directory containing config.yaml silently read
    ~/.leather/config.yaml instead, which once connected an eval to the wrong
    LLM endpoint. Load now prints a one-line stderr notice naming both files
    and the fix (--config=./config.yaml or LEATHER_CONFIG). Cwd
    auto-discovery remains deliberately absent.
  • config.Load tests are hermetic (#38) — the test suite read whatever
    sat in the developer's real ~/.leather/config.yaml (the cross-PR #34–#36
    phantom failure). Home-dir resolution is now injectable and the
    default-asserting tests isolate themselves.
  • 14-sig-triage: the battery's resume guard destroyed three finished
    cells.
    Completeness required ≥225 answered rows, but S1's mechanism is row
    loss (~220 answered every draw), so a second invocation judged its finished
    cells incomplete and re-ran them over their own archives. Completeness is now
    "250 rows + a manifest"; quality remains verify-run.sh's job. Only S1 was
    affected and its contrast still resolves — see
    eval/results/INCIDENT-s1-overwrite.md for the two draw-sets and why the
    correction is explicitly not conservative.