docs: contribution rules + security policy - #2
Merged
Conversation
Schema: add step_id, evidence_target{subject,question}, machine-readable
satisfaction{requires_nonempty,requires_relevance_to,freshness}; ordered_route
now references step_ids; structured fallback{provider,when}; add schema_version,
contract_id, abstain_reason; rename source→source_scope. Split execution result
into schema/use-receipt.schema.json (obligation vs result) — makes gate-gaming
detectable, not just 'tool returned something'.
Tells: tighten hist-did-we (require decide/ship/session verb, not bare 'did we')
and runtime-alive (require runtime noun/qualifier); add per-family terminal
tells; clause-scoped negation (negatives no longer a stub).
Corpus: split dev-corpus (tuning) vs frozen eval-corpus (never tune against);
add paired minimal cases, reversed compounds, clause-scoped negation, abstain.
Add satisfaction-cases.jsonl for honest gate-gaming simulation.
Base for dae/contrib engine build (decompose + tells matcher + benchmark runner).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… benchmark section Lead hook + evidence-family framing, before/after, ASCII pipeline, one-command reproduce, a Compared-to table vs semantic-router, and a radically honest Benchmark section: labels the N=26 corpus as synthetic/self-labeled (NOT proof on real work) and spells out the 3 real tests still owed (real transcript corpus, live shadow run, live gate-gaming). Mirrors ponytail's benchmark-honesty ethic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ement - src/embed.js: local bge-m3 via Ollama (/api/embed), no API key; fail-open probe. - policy/exemplars.yaml: diverse per-family exemplars + explicit abstain class (written from rubric semantics, NOT the frozen real eval — integrity). - src/semantic.js: nearest-exemplar classifier, disk-cached exemplar embeddings, abstain-class + score-floor + margin gates (won't route everything). - planner: shared buildStep/assemble; sync plan() (tells) unchanged + async planSemantic() (tells → semantic fallback), fail-open on embed error. - scripts/bench-semantic.js: tunes on dev only, evaluates once on frozen real corpus. Honest real-corpus result: tells-only 1/85 routed → +semantic 3/85 with the conservative dev-picked threshold. DIAGNOSTIC (no threshold): bge-m3 puts the correct family in top-1 57.6% / top-2 83.5% of the 85 routed cases — the signal is real; the blocker is the abstain class swallowing terse routing prompts (abstain outscores the true family on 35/85). Next: better route-vs-abstain calibration, or a local-LLM zero-shot fallback. 37/37 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ot a 5th class Argmax over the 4 evidence families, then a deterministic route gate: route iff top family clears a per-family floor AND beats runner-up by margin (or top-k neighborhood agrees) AND no abstain-veto. Abstain-exemplar proximity is an optional veto signal, not an argmax competitor — fixes generic abstain exemplars swallowing terse routing prompts. Emits Dae's diagnostic fields (candidate_family, family_score, runner_up, family_margin, abstain_score, topk_agreement, route_gate, abstain_reason). Eval-1 discovery/regression (NOT a final claim — the 250 is now design-contaminated per Dae; a fresh blind Eval-2 comes after calibration): first-family-correct 1 -> 14 (floor .60, abstain-recall 83%) up to 37 (floor .50). Coverage-risk curve now tunable. Next: session-disjoint calibration set → freeze floors/margins/veto → blind Eval-2. 37/37 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
200 real prompts, session+cluster-disjoint from Eval-1 (the 250) AND from the held Eval-2 set. Group-split by session (whole sessions to one bucket, per Dae's anti- sibling-leakage rule). Secrets scrubbed; 0 post-scrub flags. Fields: id, prompt, source_hash, cluster_id, session_hash. For calibrating per-family floors/margins/ veto ONLY — never the eval sets. Eval-2 (200, also disjoint) is held until the calibration config is frozen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
200 real prompts, session+cluster-disjoint from Eval-1 AND the calibration set. Held until Dae locked the Stage 2.1 config (lock f938561...). For blind labeling + a single scored run — the clean final scorecard. Fields: id, prompt, source_hash, cluster_id, session_hash. No labels. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rubric-following classifier via local qwen (Ollama, no API key, prompts stay on-box), pluggable backend, same classify(text) contract as the embedding classifier so it drops into the planner. On calibration (dev, N=200, gold-labeled): first-family recall 35/76 = 46.1% vs the deterministic gate's 4/76 = 5.26% (~9x), routed precision 74.5%, abstain-recall 91.1%. Untuned (raw rubric, no few-shot). This validates the LLM path as the product's real routing engine; the embedding gate is the honest floor it beats. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esis) Smallest real version per the S382 reframe (Dae + Fable converged): vendor-neutral evidence-discipline layer. 5 components (claim-obligation detector, evidence receipts, narrow finalization enforcement, authority policy, router-as-advisory), reuse map, dogfood-on-Pingu → 3-external-teams go/no-go, explicit kill-gate, and the 10 hardening blockers carried from Dae's Eval-2 review. Routing is advisory, not the pitch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…al transcripts
src/claims.js: deterministic high-risk-claim detector (+ self-check). scripts/
dogfood-transcripts.js: runs it over Pingu's own Claude Code transcripts, checks each
claim against recent-window evidence, flags unsupported ones. Safe (read-only, no
live-env change).
FINDING (80 sessions, 68 claims): keyword claim-detection over-fires — 'is live/
running/healthy' match conversational uses ('is live — his reply wakes me', 'running
in the background', quoting 'the code does X'). Naive detection has too many false
positives to enforce on — the claim-detection half needs semantic precision (LLM),
same lesson as routing. The dogfood harness works; the detector needs the LLM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d by Fable Full Phase A build by Daedalus: hardened LLM client (digest-pinned qwen), LLM-assisted claim-detector (keyword candidates -> LLM confirm, rejects quoted/conversational), authority.yaml (shadow), claim-support receipts (tamper-detect, 0600 append-only), shadow finalization (always log_only). 86/86 tests before a final receipt provenance fix (wrong-source now carries actual provider + authority_tier:null) that was NOT yet re-run/reviewed. Preserved to origin because Dae hit iteration budget mid-fix. NOT a completion claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
createClaimSupportReceipt assigned the configured authoritative tier even when the contacted source was wrong; validateReceipt (correctly) requires authority_tier:null for failure=wrong_source. Set tier to null on wrong_source. Completes the mid-fix Dae was on when his iteration budget cut off. Full suite 86/86. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lem solved
Runs Dae's claim-detector over Pingu's real transcripts. Result (50 turns, 59 keyword
candidates): LLM CONFIRMED 33 (56%) as genuine evidence-requiring claims, REJECTED 26
(44%) as conversational/quoted false-positives. The rejected set is exactly the FPs the
keyword version wrongly flagged ('is live — his reply wakes me', quoted 'the code does
X', 'running in the background' subagents). The reframed product's precise claim-detection
works on real data; the keyword FP problem is solved. (Family classification still loose
on some — a tuning item, not a blocker.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(Dae fix) Fixes the blocking architectural flaw Dae found at 4ad1079: receipts required a claim_id at PostToolUse time, but the claim/entity don't exist until Stop. Splits into two causally-honest artifacts: - src/evidence.js (new): evidence-contact receipt emitted at PostToolUse — no claim needed. Carries session/turn/tool-call IDs (fixes cross-turn leak), args_sha256 + arg_keys only (never raw args → no secret leak), deterministic `references` (relevance basis, not a caller boolean), failure ∈ {none,empty,error} (operational only). Frozen, exact-keys, canonically hashed as evidence-<24hex>. - src/finalization.js: evaluateFinalization now consumes evidence-contact receipts and materializes claim-support links at Stop — matches authoritative provider + entity in references + fresh-at-finalization → supported/unsupported/ambiguous. Shadow/log_only. Bug fixes (Dae-verified): - Future observed_at no longer counts as fresh (raw age >= 0 && <= maxAge), in both receipts.js and finalization.js. - Override no longer dead: an override forces `supported` with logged provenance regardless of contact state (was only honored when failure was already 'none'). Stage-1 scope: capability-based authority matching, multi-token entity matching, and honest replay denominators land with the offline replay harness (stage 2). Tests: 101/101 (+15) — node --test. No deploy, shadow-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ions, redaction Dae byte-reviewed 561cae7 and proved (by running the code) that the temporal split was not yet causally isolated. Closes his Stage-1.1 gate (items 1-4): 1. Session/turn scope enforced: evaluateFinalization now REQUIRES sessionId+turnId; a receipt participates only when session_id+turn_id match exactly. Out-of-scope receipts are classified `wrong_scope`, not silently dropped. (Fixes the proven session-A-evidence-supports-session-B-claim leak.) 2. Future observed_at rejected under EVERY freshness policy — the future check now binds before the `any` short-circuit, so a historical/any claim can no longer be rubber-stamped by a future observation. A future contact `timestamp` → disposition `future_timestamp`, never supporting. 3. Rich dispositions replace collapsed "unsupported": each obligation now exposes candidate_evidence_ids, supporting_evidence_ids, and per-contact dispositions with an ordered decision enum (wrong_scope → future_timestamp → wrong_authority → empty_error → entity_mismatch → stale → supporting). Stale-but-relevant evidence keeps its IDs (the audit gap Dae flagged). 4. Result-reference redaction: secret shapes (api keys, tokens, JWT, conn-string creds) are scrubbed from tool OUTPUT before token extraction, plus a high-entropy filter; new `references_redacted` count field (schema 16→17 keys). Tool output can no longer persist a credential as a reference. Unchanged (next pass, frozen before scoring): capability-based authority matching, multi-token entity matcher. Provider matching + references.includes preserved as-is. Tests: 106/106 (+5) — node --test. Shadow-only, no deploy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(Dae-approved) Replaces provider-based authority with DERIVED capability authority and adds a conservative multi-token entity matcher — the two pieces Dae required frozen before any scored replay — plus his mandatory report-scope patch. - src/capability.js (new): closed 6-capability enum; deriveCapability(provider,name,args) is deterministic from tool shape; UNKNOWN => null and null can never support a claim (Dae's core invariant). Arbitrary Bash / process-start / bare read are NOT authoritative. Versioned (cap-v0.1) + capabilityMapSha() for replay provenance. - src/entity-match.js (new): conservative matcher — atomic-identifier-exact OR all discriminative tokens (generic words like api/lambda/service never decide); records a match explanation (method, matched/unmatched terms). Versioned + entityMatcherSha(). - policy/authority.yaml v0.2: families carry tier+freshness only; capability_requirements maps each claim pattern_id to the capabilities that authoritatively satisfy it. authority.js validates the new shape; requiredCapabilities() resolves per pattern. - src/evidence.js: capability is now DERIVED internally (never a caller param); adds references_truncated so a non-match against a capped reference set becomes ambiguous, not a clean miss. Schema 17 to 18 keys. - src/claim-detector.js: obligations carry pattern_id (drives capability_requirements lookup). - src/finalization.js: capability + entity-matcher gates (wrong_capability, truncated dispositions; match_method capability_match); MANDATORY Dae patch — session_id + turn_id are recorded in the report core and hashed into enforcement_id. - src/index.js: barrel exposes the evidence-contact + capability + matcher API; drops the removed resolveAuthority. Fixes found while finishing the pass: - capability.js Bash read/health regex used a trailing-space word-boundary that failed on "grep -n" (returned null); anchored on the bare command token instead. - authority.js had 2 literal NUL bytes (an escape written as a raw NUL) — replaced with the proper escape; file is clean ASCII again. Frozen before scoring, per Dae — capability/matcher rules are not to be tuned after seeing replay results. Tests: 125/125 (node --test), incl. Dae's capability negative tests and the entity-matcher corpus. Shadow-only, no deploy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Require exit-zero runtime.test_run evidence to carry a structured, nonzero result.executed_test_count; missing or zero counts remain unknown. The normalizer deliberately does not parse free-form test output. Redact Bearer authorization credentials case-insensitively and add regressions for lowercase/mixed-case schemes.
Add seeded transcript extraction, frozen provenance validation, and coverage-only replay artifacts without scored aggregates. Executed-test counts are copied only from structured result.executed_test_count or result.executedTestCount fields; free-form output is never parsed.
The frozen replay.js summary emits 'evidence_contact_count'; the LL-3 scorer's CLI validation expected 'evidence_contacts' (three occurrences), rejecting every genuine coverage summary. Per-row validation already used the correct key; test fixtures already used the correct key — only the summary-level strict-shape path was wrong and no end-to-end test fed it a real replay summary. Orchestrator trivial-compat fix (frozen producer wins). 159/159. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… frozen replay) Completes the previous fix: replay.js also emits harness_version in the coverage summary; the scorer's exactKeys list omitted it. Key set now proven identical to the frozen producer's output. Validates harness_version === replay-v0.1. 159/159. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scorer required the labeling summary's recorded runtime to equal the current validated runtime verbatim — including the scorer's OWN blob at label time. Labels never execute the scorer, so scorer-side fixes (like the two coverage-shape compat fixes) would force a full $3.50/5h relabel for provenance theater. New binding: every labeler-execution blob (label-corpus.js, exam-label schema, frozen replay/ detector set) must match byte-for-byte; scorer-only artifacts (score-exam.js, exam-score.schema.json) are shape-checked but excluded from label-time equality — the scoring runtime remains separately attested in the score output. Path sets must still match exactly. 159/159. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- README: banner hero, badges, real acceptance-transcript demo, evidence-family table, 60-second dry-run start, policy packs with burn receipts, honesty section with live bench numbers (62.5% first-route N=18, 0/4 gaming bypasses) - AGENTS.md: how agents should behave under Leadline enforcement - llms.txt: documentation map for LLM crawlers - brand/: banner.svg + demo.svg (S382 identity system) - package.json: dense description + 14 defensible keywords Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- badge: claude-code-hooks (not plugin — no marketplace manifest yet) - scope trace claim to enforcement decisions; families framed via routes.yaml - soften absolute receipt claim; llms.txt: supported obligations phrasing - install: npx github:Nazim22/leadline (works today) until npm publish Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- CONTRIBUTING.md: test discipline, burn-receipt rule for packs, issue-first surfaces, exam-protocol change rules (fresh-slice only), claims-with-receipts house style, AI-attribution convention - SECURITY.md: private vulnerability reporting path, honest threat model (local-first single-operator; hostile same-UID backlog disclosed via EXAM.md) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CONTRIBUTING.md (gbrain-style: test discipline, burn-receipt rule for packs, issue-first for engine/security surfaces, fresh-slice rule for exam changes) and SECURITY.md (private vulnerability reporting, honest threat model with the EXAM.md hardening backlog disclosed).
🤖 Generated with Claude Code