dogfooding: enforce + test the validators the toolkit ships#4
Merged
Conversation
…sed slugs The counter-atomicity check read the YEAR of DR-/CF-YYYY-MM-DD-... slugs as a counter, flagging every same-year date-based DEC/CF as a collision. SliceOps uses both counter-based (INS-/LP-/HANDOFF-/DEC-NNN) and date-based (DR-/CF-YYYY-MM-DD) naming; the check now excludes date-based slugs and only collides true counters. Surfaced by the first-ever run of validators.py against the canonical vault corpus (2026-06-19). Per DR-2026-06-19 dogfooding-enforcement-mandate (calibrate the check, do not suppress) + LP-003 (published-not-enforced). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the repo's central contradiction named in the 2026-06-19 audit (§2): "a validator toolkit with no test of its validators." - tests/: stdlib regression suite pinning the documented false-positive fixes (date-slug, band-aware principle count, band-unit anti-pattern guard, singular-Entity title) + positive detections for real defects - ci.yml: run the unittest suite + validators self-application (corpus-agnostic checks) + gitleaks secrets scan (P7 Security-by- Construction); keep B.2 concurrency-cancel + draft-skip + aggregation-gate Authorizing decision: DR-2026-06-19-dogfooding-enforcement-mandate (LP-003 published-not-enforced: a prescribed gate with no executing workflow is a build failure, not a header comment). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
anrasi
added a commit
that referenced
this pull request
Jun 22, 2026
Addresses the 2026-06-19 external review. Each fix ships with regression tests (33 total; green with PyYAML and with the stdlib fallback CI uses). validators.py: - #2 Windows/Linux path portability: OS-agnostic part checks (_norm_parts) for frozen-dir skip, the .git skip, and .github/workflows detection (was '/'-only substring matching → broke on Windows os.walk '\\'). - #3 topic-tags semantics: unconfigured --topic-taxonomy now SKIPPED (green), configured-but-missing is a clear hard error (was a confusing "skipped" that failed the build). - #4 frontmatter parser: uses PyYAML automatically when installed (robust: block scalars, nested, comments); documented stdlib subset + inline-comment handling as fallback. No dependency required. - #5 llm-ci-cost: strips YAML comments before the endpoint heuristic (no more false-positive on comments/docs); structural concurrency + on.pull_request checks via PyYAML when available (handles the on:->True YAML quirk); custom endpoint/proxy limitation documented. calibrate.py: - #6 propose_bands ignored its percentile inputs and returned fixed bands → split into canonical_bands() (fixed reference) + observed_bands(p) (genuinely data-driven from p25/p50/p75/p90); output now prints both, honestly labeled. - #7 percentiles used exclusive method (extrapolated past observed max on small samples) → inclusive + clamp to [min,max] + n==1 handling + small-sample (<8) advisory warning. #1 (counter-atomicity date-slug false-positive) was already fixed in v0.1.0 (toolkit#4) — the review ran the pre-fix 2026-06-19 snapshot. CI self-run upgraded to --checks all (topic-tags SKIPs cleanly now). New tests/test_calibrate.py adds the first calibration tests. Per DR-2026-06-19 (gates + tests for every fix). SL-010. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the toolkit's "recursive dogfooding" claim true.
Per
DR-2026-06-19-dogfooding-enforcement-mandate(audit 2026-06-19 §2: "a validator toolkit with no test of its validators";LP-003published-not-enforced).What lands
tests/— stdlibunittestregression suite (Py 3.9+, no deps) pinning the documented false-positive fixes (date-slug, band-aware principle count, band-unit anti-pattern guard, singular-Entity title) + positive detections.ci.yml— now runs: the test suite +validators.pyself-application (corpus-agnostic checks) + gitleaks secrets scan (P7). Keeps B.2 concurrency-cancel + draft-skip + aggregation-gate.DR-2026-…slugs no longer false-collide) — required so the spec CI (which fetches this validator) goes green.Evidence
python3 -m unittest discover -s tests -v→ 12 passedvalidators.py --root . --checks counter-atomicity,principle-count-coherence,entity-count-coherence,band-unit,llm-ci-cost→ exit 0