Add MDS037 duplicated-content rule for cross-file paragraph detection

[jeduden]

Summary

Implements MDS037, a new opt-in linting rule that detects paragraphs duplicated verbatim across Markdown files in a project. The rule uses SHA-256 fingerprinting over normalized text (lowercase, whitespace-collapsed) to identify substantial duplicate content.

Key Changes

• New Rule Implementation (internal/rules/duplicatedcontent/rule.go):

  • Fingerprints paragraphs using SHA-256 over normalized text (lowercase, single-space-separated, trimmed)
  • Configurable min-chars threshold (default 200 runes) to avoid noise from short boilerplate
  • Scans corpus via RootFS when project root is known, falls back to file's directory
  • Supports include and exclude glob patterns (matched against slash path and basename) to scope the cross-file comparison
  • Opt-in by default (EnabledByDefault() == false) since projects intentionally share prose across files
  • Uses the shared settings.ToInt / settings.ToStringSlice helpers (plan 85)

• Comprehensive Test Suite (internal/rules/duplicatedcontent/rule_test.go) — 97%+ coverage, including CLI-style relative f.Path under an absolute RootDir, basename-only exclude across directories, and .. traversal rejection.

• Documentation (internal/rules/MDS037-duplicated-content/README.md):

  • Explains the normalization strategy and min-chars threshold
  • Documents the O(N²) read cost and recommends an exclude entry for generated directories
  • Correctly labels the rule as disabled by default (opt-in via .mdsmith.yml)

• Test Fixtures: single-file good fixture; a bad/duplicate.md + bad/ref/source.md pair that shares neutral wording (no self-references) to avoid confusing future readers.

• Integration: rule registered in cmd/mdsmith/main.go and test imports; rules index and plan 50 status updated.

Also in this PR

• Plan 90 (plan/90_corpus-test-git-config-isolation.md): a small follow-up plan filed in the same session to track isolating internal/corpus/clone_test.go:makeBareRepo from host-level git commit signing, which surfaced as a pre-existing test failure while verifying this PR. The plan itself is documentation only; no code change.

Implementation Details

• Paragraphs are extracted from AST and normalized by collapsing whitespace, lowercasing, and trimming
• Fingerprints are deterministic SHA-256 hashes enabling efficient duplicate detection
• Corpus indexing is performed once per file check, with matches sorted for deterministic diagnostics
• Files that fail to read or parse are silently skipped (advisory rule, never fails a run)
• Diagnostics report the line in the current file and the path/line of the duplicate in another file

https://claude.ai/code/session_018U79bfk1Gde7UHt9M7Ycm3

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.68%. Comparing base (b76ed84) to head (14ef6d9).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #154      +/-   ##
==========================================
+ Coverage   87.63%   87.68%   +0.04%     
==========================================
  Files         108      108              
  Lines       13597    13597              
==========================================
+ Hits        11916    11922       +6     
+ Misses       1222     1218       -4     
+ Partials      459      457       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds a new opt-in mdsmith rule (MDS037) to detect verbatim duplicated paragraphs across Markdown files, along with docs, fixtures, and registration so it can be enabled via config when desired.

Changes:

• Implement MDS037 duplicated-content rule using normalized paragraph SHA-256 fingerprints and cross-file corpus scanning.
• Add unit tests plus folder-based integration fixtures and rule documentation.
• Register the rule in CLI/test imports and update the rule index + plan status.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
internal/rules/duplicatedcontent/rule.go	New rule implementation: paragraph extraction, normalization, corpus scanning, include/exclude filtering, and diagnostics.
internal/rules/duplicatedcontent/rule_test.go	Unit tests for detection, normalization, settings, and error handling.
internal/rules/MDS037-duplicated-content/README.md	New rule documentation, configuration options, and examples.
internal/rules/MDS037-duplicated-content/good/simple.md	“Good” fixture content for the rule docs/fixtures.
internal/rules/MDS037-duplicated-content/bad/duplicate.md	“Bad” fixture declaring expected diagnostic for a duplicate paragraph.
internal/rules/MDS037-duplicated-content/bad/ref/source.md	Reference file used as the duplication source during fixture runs.
internal/rules/index.md	Add MDS037 to the rules index.
cmd/mdsmith/main.go	Register rule via blank import.
internal/integration/rules_test.go	Register rule for fixture-based integration test runs.
internal/engine/categories_test.go	Register rule for category-related tests.
internal/config/config_test.go	Register rule for config-related tests.
plan/50_redundancy-duplication-detection.md	Mark plan complete and summarize implementation decisions.
PLAN.md	Update plan index status for item 50.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

[jeduden]

🟢 Merge Queue — picked up

This PR is in the queue and will be batched with other queue-labelled PRs.

Next: No action needed — you'll get another comment when CI starts on the batch. View merge queue run.

[jeduden]

🔵 Merge Queue — CI running

Merged into batch branch merge-queue/batch-154-1776847981 alongside #158. View CI run.

Next: No action needed — you'll be notified when CI completes.

[jeduden]

✅ Merge Queue — merged

This PR landed on main via commit 212bf1d. CI run that validated the merge.

Next: Done — nothing more to do here.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

[jeduden]

🟢 Merge Queue — picked up

This PR is in the queue and will be batched with other queue-labelled PRs.

Next: No action needed — you'll get another comment when CI starts on the batch. View merge queue run.

[jeduden]

🔵 Merge Queue — CI running

Merged into batch branch merge-queue/batch-154-1776875561. View CI run.

Next: No action needed — you'll be notified when CI completes.

[jeduden]

✅ Merge Queue — merged

This PR landed on main via commit deea242. CI run that validated the merge.

Next: Done — nothing more to do here.