You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
15-trust-repair pre-pilot scaffold — arms, runner, and oracle wiring
for the trust-boundary-repair workload (decision of record for the second
workload; full corpus and confirmatory runs remain gated).
Changed
14-sig-triage: the ablation findings are now pre-registered and
replicated. Six contrasts were frozen at main commit 96cc418 before any
confirmatory cell ran; the battery executed 11 arms × 3 draws (5× on the one
contrast that hit the registered boundary trigger), and five of six
survive Holm–Bonferroni at α=0.05. Published figures move accordingly, and
every movement is against the author's interest: decomposition depth from
−9.2 to −5.2, retrieval payload from +6.4 to +3.0, and retrieval
payload further from RESOLVED to UNRESOLVED under Amendment 2. The
headline span widens to 59.6→81.6% because a registered S1 draw is now
the lowest archived cell. New rendered page: eval/results/CONFIRMATORY.md.
14-sig-triage: the confirmatory estimator is now issue-clustered.
Amendment 2 replaces pooled McNemar as the primary test. Pooling concatenated
repeated measurements of the same 250 issues and treated them as independent
trials, which overstates significance; the primary is now a sign-flip
permutation test over whole issues. Effect sizes are unchanged — only the
uncertainty was wrong. Cost: contrast 2 (retrieval payload) drops to
unresolved. Pooled McNemar remains in the output, labelled descriptive.
14-sig-triage: "a bad harness scores below no harness at all" is
retired. Across replication the fresh-session scheme (61.3%, 5 draws) and
the bare model (61.9%, 4 draws) are statistically level. The claim is now
that a bad harness ties the bare model while paying for an extra stage and
250 tool calls — which is the stronger statement anyway.
Fixed
temperature: 0 is finally reachable from a single setting (#56) —
greedy decode required setting temperature in bothconfig.yaml and every
agent's frontmatter, because the frontmatter parser defaulted to 0.7 (which
shadowed config) while resolveAgent used 0 as its unset sentinel (which
made an explicit agent-side 0 read as "unset"). Set-ness is now tracked
through frontmatter and lifecycle YAML (TemperatureSet, mirroring EnabledSet), the built-in 0.7 lives in exactly one place
(config.DefaultTemperature), and the documented priority lifecycle >
frontmatter > config.yaml > built-in actually holds. The agent-config debug
line logs temperature_source=agent|config so this class of shadowing is
visible, and leather doctor gained a temperature row.
leather doctor no longer fabricates config problems (#31) — source
attribution used a value != default heuristic, so a value explicitly set
equal to the built-in default was labelled default, and the catch-all config/env/flag never said which layer won. config.Load now records true
per-key provenance and doctor reports it: yaml, env, flag, or default. Also fixed en route: LEATHER_PERSIST_RUNS_DETAIL and LEATHER_PERSIST_RUNS_TOOL_CAP were only read at startup seeding, so a YAML
value silently beat those two env vars, contradicting the documented order.
The documented config precedence was wrong, not the code (#33) — docs/modules/config.md claimed YAML overrides environment variables; the
code has always applied env over YAML. The doc now states the real order — flags > env > YAML > built-in — plus the config-file resolution chain, and
a test pins env-over-YAML for llm_endpoint, the field from the original
wrong-endpoint incident.
Falling back past a project-local config.yaml is now loud (#30) —
invoking leather bare in a directory containing config.yaml silently read ~/.leather/config.yaml instead, which once connected an eval to the wrong
LLM endpoint. Load now prints a one-line stderr notice naming both files
and the fix (--config=./config.yaml or LEATHER_CONFIG). Cwd
auto-discovery remains deliberately absent.
config.Load tests are hermetic (#38) — the test suite read whatever
sat in the developer's real ~/.leather/config.yaml (the cross-PR #34–#36
phantom failure). Home-dir resolution is now injectable and the
default-asserting tests isolate themselves.
14-sig-triage: the battery's resume guard destroyed three finished
cells. Completeness required ≥225 answered rows, but S1's mechanism is row
loss (~220 answered every draw), so a second invocation judged its finished
cells incomplete and re-ran them over their own archives. Completeness is now
"250 rows + a manifest"; quality remains verify-run.sh's job. Only S1 was
affected and its contrast still resolves — see eval/results/INCIDENT-s1-overwrite.md for the two draw-sets and why the
correction is explicitly not conservative.