Skip to content

Any two PRs that add tests conflict on six files by construction (test count hard-coded) #293

Description

@cdeust

Problem

The collected test count is hard-coded, by hand, in six files: .bestpractices.json, CLAUDE.md, CONTRIBUTING.md, README.md, docs/ASSURANCE-CASE.md, assets/badge-tests.svg. Each is checked for exact equality against whichever branch's own live pytest --collect-only count runs in CI.

That count is a property of the post-merge tree, not of any one branch: two branches that each add tests compute two different, both-true numbers and must each edit the same six lines to match. Whichever merges second silently overwrites the first's correct figure with its own now-stale one.

Measured on this repo, not hypothesised:

Fix (this issue tracks the PR that resolves it)

  • assets/badge-tests.svg becomes the ONLY artifact stating an absolute test count.
  • Its check moves from exact equality to a monotone floor (committed <= live) in both scripts/check_doc_claims.py (doc_claim_structural.check_badge_floor) and scripts/generate_repo_badges.py (stale_tests_badge) — a committed count that lags the live one is stale-but-true and passes; only an over-claim fails.
  • The other five files drop the number entirely, pointing at the badge instead.
  • A standing regression test (RepositoryTests.test_no_prose_file_states_the_suite_size_any_more) asserts no scanned file — .bestpractices.json included — states this claim in prose again.
  • scripts/check_doc_claims.py (420 lines) and scripts/generate_repo_badges.py (305 lines) were both over the repo's 300-line file cap before this work needed to touch them further; split via Extract Module into doc_claim_sources.py/doc_claim_scan.py/doc_claim_structural.py/repo_badge_catalog.py, verified behavior-preserving by an unchanged test suite before the floor logic was added.

Acceptance test: two throwaway branches off the fix branch, each adding a trivial passing test, merge with zero conflicts in the six files.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions