Skip to content

Milestone 18: Bootstrap doctor/audit contract test expansion#15

Merged
incredincomp merged 3 commits intomainfrom
copilot/expand-doctor-audit-contract-tests
Mar 14, 2026
Merged

Milestone 18: Bootstrap doctor/audit contract test expansion#15
incredincomp merged 3 commits intomainfrom
copilot/expand-doctor-audit-contract-tests

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 14, 2026

Adds explicit contract tests for all key target-repo diagnosis semantics in bootstrap_doctor.py, which were previously untested despite being stable operator-facing contracts (Milestone 16). Proves classification stability and anti-drift for the doctor layer.

New: tests/test_bootstrap_doctor.py (77 tests across 12 test classes)

  • All six health state classifications via classify_health() — including priority ordering (stale-version beats profile-mismatch), patch-only version differences not triggering stale state, and low-confidence suggestions not triggering profile mismatch
  • Version comparison helpers_semver_tuple() and _is_materially_behind() unit-tested directly
  • Status helpersmarker_status(), required_files_status(), placeholder_status(), profile_alignment() (all 4 alignment states)
  • recommend_actions() for every health state — including an explicit assertion that --force never appears in any recommendation
  • Era classification alignment — proves bootstrap_doctor.classify_era() always agrees with bootstrap_core.classify_marker_era()
  • audit() integration tests using real temporary directories for 5 key states; includes a read-only proof that audit() does not mutate the target directory
  • Health state name stability contract — asserts HEALTH_LABELS and HEALTH_DESCRIPTIONS cover exactly the six documented states

Integration tests use a self-contained fake source root with a pinned VERSION file (_TEST_MARKER_VERSION = "0.14.0") so tests are version-independent and don't break on future version bumps.

Infrastructure updates

  • tests/test_bootstrap_doctor.py added to BOOTSTRAP_REPO_REQUIRED_FILES in validate_bootstrap.py and bootstrap-manifest.yaml
  • AGENTS.md — expanded "Bootstrap doctor" rules section with anti-drift contract: diagnostic semantics are test-backed, output phrasing can vary but semantic meaning must not, shared helpers (classify_era, parse_marker) require updates to both test files when changed
  • README.md — updated layout tree and Contract tests section to document doctor test coverage and the conservative-behavior guarantees the tests prove
  • VERSION bumped 0.14.0 → 0.15.0; CHANGELOG.md updated

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits March 14, 2026 09:42
…cker

Co-authored-by: incredincomp <43874843+incredincomp@users.noreply.github.com>
…d-and-healthy test self-contained

Co-authored-by: incredincomp <43874843+incredincomp@users.noreply.github.com>
Copilot AI changed the title [WIP] Add contract tests for target repo bootstrap doctor and audit Milestone 18: Bootstrap doctor/audit contract test expansion Mar 14, 2026
Copilot AI requested a review from incredincomp March 14, 2026 09:45
@incredincomp
Copy link
Copy Markdown
Owner

@copilot
verify before merging:

the tests are asserting semantic meaning, not overfitting to phrasing or incidental implementation details; the summary suggests that is already the intent.

CI/checks actually run cleanly once GitHub finishes loading them; the page showed some GitHub “Uh oh!” load errors in the conversation view, so I could read the Copilot summary but not rely on the page for a perfect end-to-end check-status view.

@incredincomp incredincomp marked this pull request as ready for review March 14, 2026 09:47
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 14, 2026

@copilot
verify before merging:

...

Verified both items:

CI: clean. The CI run on the latest commit (b60511f) completed with success — py_compile, validate_bootstrap, 120 contract tests, and the fixture self-test all passed.

Tests: semantic meaning, not phrasing. A few specifics:

  • TestClassifyHealth / TestClassifyEra / TestProfileAlignment / TestMarkerStatus etc. all assert on return-value strings from semantic functions (e.g., "populated-and-healthy", "aligned", "pre-version"), not on human-readable label or description text.
  • TestRecommendActions asserts which tool appears in the recommendations (suggest_profile.py, refresh_bootstrap.py, etc.) and which flags are required (--dry-run) or forbidden (--force) — i.e., the operator-safety contract — not exact command phrasing.
  • TestHealthStateNames only asserts that HEALTH_LABELS.keys() and HEALTH_DESCRIPTIONS.keys() equal the set of six documented state names. It does not inspect the human-readable label values themselves, so renaming "Unbootstrapped" to something else would not break these tests.
  • The integration TestAuditIntegration tests use a fake source root with a pinned VERSION file so they're isolated from future version bumps.

No tests are coupled to output formatting or incidental implementation details.

@incredincomp incredincomp merged commit 46e9a53 into main Mar 14, 2026
5 of 6 checks passed
@incredincomp incredincomp deleted the copilot/expand-doctor-audit-contract-tests branch March 14, 2026 09:48
Copilot stopped work on behalf of incredincomp due to an error March 14, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants