fix(baseline): widen pattern file lists to accept README.rst / .txt / no-ext variants - #407
Open
mlieberman85 wants to merge 1 commit into
Open
fix(baseline): widen pattern file lists to accept README.rst / .txt / no-ext variants#407mlieberman85 wants to merge 1 commit into
mlieberman85 wants to merge 1 commit into
Conversation
… no-ext Follow-up to darnitdevorg#402 (which was TOML-only inside `llm_eval` blocks). Widens the sibling `pattern` handler's `files` list on the same 5 controls that gate an `llm_eval` pass, so the deterministic tier resolves PASS/FAIL conclusively on more repos rather than falling through to the LLM tier. The bug pattern before this change: a Sphinx-based Python project that ships README.rst but no README.md would have `pattern.files` scan zero on-disk candidates, return INCONCLUSIVE via `_regex_no_files_result`, and the pipeline would fall through to `llm_eval`. This was the exact trigger for the empty-file_contents consultation surveyed across 29 repos (4 on tqdm, 5 on flask when its README.md was renamed to README.rst). Controls widened: - OSPS-DO-03.02 (VerifyReleaseAuthor): + README.rst / README.txt / README - OSPS-DO-04.01 (HasSupportDocs): + README.rst / README.txt / README + docs/support.md - OSPS-DO-05.01 (SupportEndOfLife): + README.rst / README.txt / README - OSPS-SA-01.01 (ArchitectureDocs): + rst variants for ARCHITECTURE/DESIGN/README/docs - OSPS-SA-02.01 (ApiUsageDocs): + rst variants for API/README/USAGE + docs/api.rst `_regex_match_files` reads each listed file in full; the additions stay well under the handler's `_FILE_SCAN_LIMIT = 100` cap and the per-file 2000-byte truncation. Regression guard: `tests/darnit_baseline/controls/test_pattern_file_variants.py` - Parametrized across the 5 gated controls; asserts README.rst is present in each pattern list. - Also asserts README.txt / README (no ext) on the DO-* controls where they were added. Not in scope: framework code changes to the pattern handler's file resolution or generalizing to auto-derive .rst/.txt variants from .md entries. Both are separate.
3 tasks
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #402 (currently in review as #406). Where #406 was TOML-only inside
llm_evalblocks, this widens the siblingpatternhandler'sfileslist on the same 5 controls that gate anllm_evalpass, so the deterministic tier resolves PASS/FAIL conclusively on more repos rather than falling through to the LLM tier.Motivation
The bug pattern that #402 fixed downstream: a Sphinx-based Python project that ships
README.rstbut noREADME.mdwould have thepattern.fileslist scan zero on-disk candidates, return INCONCLUSIVE via_regex_no_files_result, and the pipeline would fall through tollm_eval. #406 makes thatllm_evalfallback useful (real file content); this PR makes the fallback fire less often by covering rst/txt/no-ext README on the deterministic tier.Controls widened
README.rst,README.txt,READMEREADME.rst,README.txt,README,docs/support.mdREADME.rst,README.txt,READMEARCHITECTURE.md,DESIGN.md,README.md, docs pathsAPI.md,README.md,USAGE.md,docs/api.md_regex_match_filesreads each listed file in full; the additions stay well under the handler's_FILE_SCAN_LIMIT = 100cap and the per-file 2000-byte truncation.Test plan
tests/darnit_baseline/controls/test_pattern_file_variants.py(new, 8 parametrized tests): asserts README.rst is present on every gated control's pattern list; asserts README.txt / README (no ext) on the DO-* controls where they were added.pytest tests/darnit_baseline/ -q-> 877 pass, 7 skip.pytest tests/ -q-> 3010 pass, 26 skip, 0 fail.ruff check .clean (repo-wide, not just touched files).python scripts/validate_sync.py --verbose-> TOML schema valid, 66 controls, 7 handlers in sync.Relationship to #406
Complementary:
fix-402-llm-eval-file-contents): fixes what the LLM tier sees WHEN it fires. TOML-only insidellm_evalblocks.widen-pattern-file-lists): fixes how often the LLM tier fires. TOML-only insidepatternblocks.Either can merge independently. Together they close the tqdm-style failure mode from both sides: the pattern tier is more likely to resolve conclusively, and when it doesn't the LLM tier now sees real content.
Not in scope
Both are separate follow-ups.