Preserve fenced code block contents during wrapping (#329)

[lodyai]

Summary

This branch preserves fenced code block contents during the combined wrapping and post-processing pipeline for issue #329. It makes Markdown tokenization use the shared fence tracker, so nested literal fences remain inside their outer fenced block and --ellipsis cannot rewrite examples after --wrap has run.

Closes #329.

Review walkthrough

• Start with src/wrap/tokenize/mod.rs to see the tokenizer move from boolean fence state to the shared FenceTracker semantics.
• Then review tests/wrap/cli.rs for the combined --wrap --renumber --breaks --ellipsis --fences regression covering SQL, JSON-like examples and nested literal fences.

Validation

• make test BUILD_JOBS='--test wrap test_cli_wrap_fences_ellipsis_preserve_fenced_content': passed.
• coderabbit review --agent: passed with 0 findings.
• make check-fmt: passed.
• make lint: passed.
• make test: passed.

Summary by Sourcery

Preserve fenced code block contents during Markdown wrapping by aligning the tokenizer’s fence handling with the shared fence tracking semantics used elsewhere.

Bug Fixes:

• Ensure combined wrapping and post-processing flags (including --wrap, --ellipsis, --renumber, --breaks, and --fences) do not modify the contents of fenced code blocks, including nested literal fences.

Documentation:

• Document the shared fence tracking behavior for tokenize_markdown so that fenced code blocks are consistently preserved and post-wrap transforms do not alter their bodies.

Tests:

• Add regression and snapshot tests for the combined CLI flags to verify fenced content is preserved for realistic SQL and JSON-like examples and nested fences.
• Add property-based tests that generate various fenced code blocks to assert their bodies are unchanged by the combined flags.
• Expand tokenize_markdown unit tests to cover closed fences, nested fences, malformed fences, and post-fence prose tokenization.
• Enable the new tokenize_markdown test module in the wrap test suite.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9ef512fa-1666-402a-8ff6-18d20ebf06ef

📥 Commits

Reviewing files that changed from the base of the PR and between 51aa44c and 74f0742.

⛔ Files ignored due to path filters (1)

• tests/snapshots/wrap__wrap_integration__cli__issue_329_wrap_fences_ellipsis_preserve_fenced_content.snap is excluded by !**/*.snap

📒 Files selected for processing (4)

• docs/developers-guide.md
• tests/wrap/cli.rs
• tests/wrap/cli_issue_329_property.rs
• tests/wrap/mod.rs

---

Summary

This PR resolves issue #329 by implementing shared FenceTracker usage in the Markdown tokenizer to preserve fenced code block contents during combined wrapping and post-processing operations.

Problem

The reported issue prevented mdtablefix from corrupting documentation examples when applying combined post-processing flags (e.g., --wrap, --ellipsis, --fences, --renumber, --breaks). Previously, these transformations could alter lines inside fenced code blocks (SQL, JSON, Rust, Mermaid diagrams), producing syntactically incorrect examples and unsafe documentation diffs.

Solution

The tokenizer in src/wrap/tokenize/mod.rs was refactored to use FenceTracker (the same fence-tracking implementation used by wrap_text), replacing the previous in_fence boolean toggle. Lines are now routed through fence_tracker.observe(line), which determines fence boundaries and drives Token::Fence emission. Content inside fenced blocks is preserved verbatim until the matching closing marker is encountered.

Changes

Core Implementation:

• Updated src/wrap/tokenize/mod.rs to integrate FenceTracker, replacing local fence-toggle logic
• Added module-level documentation describing fence tracking semantics and how nested literal fences remain intact
• Updated docs/developers-guide.md to document FenceTracker usage in tokenization and clarify that post-wrap transforms cannot alter fenced block bodies

Testing:

• Added comprehensive unit tests in tests/wrap/tokenize_markdown.rs covering closed fences, multi-line fenced blocks, nested fences, and malformed fence handling
• Added regression test ISSUE_329_COMBINED_FLAGS in tests/wrap/cli.rs using snapshot testing to validate the combined-flag scenario
• Added property-based tests in tests/wrap/cli_issue_329_property.rs with fence generators and locators to verify fence-preservation invariants across random fenced block inputs
• Created test fixtures: tests/data/issue_329_wrap_fences_ellipsis_input.txt, issue_329_wrap_fences_ellipsis_sql_body.txt, issue_329_wrap_fences_ellipsis_json_body.txt, and issue_329_wrap_fences_ellipsis_inner_fence.txt

Validation:
All automated checks reported passing: targeted regression tests, snapshot tests, property-based tests, and linting (make check-fmt, make lint, make test).

Walkthrough

Refactor the markdown tokenizer to use the shared FenceTracker for fence detection and emission; expand unit tests and add CLI regression plus property tests and fixtures to ensure combined-flag runs do not mutate fenced code block bodies.

Changes

Fenced Content Preservation During Wrapping

Layer / File(s)	Summary
Fence-state tracking refactor src/wrap/tokenize/mod.rs, docs/developers-guide.md	tokenize_markdown replaces the local in_fence boolean and is_fence(line) toggle with super::FenceTracker. Initialise fence_tracker, call fence_tracker.observe(line) to detect fence delimiter lines and use fence_tracker.in_fence() for subsequent-line checks while preserving Token::Fence emission and newline-token logic. Developer guide documents the preserved fenced-region guarantee.
Unit tests for tokenize_markdown tests/wrap/tokenize_markdown.rs, tests/wrap/mod.rs	Add and adjust tests for closed/unclosed fences, multi-line fenced bodies, nested fences and literal inner fences, malformed delimiter tokenisation, and ensure text after fences is tokenised as normal. Register tokenize_markdown test module.
CLI regression and property tests + fixtures tests/wrap/cli.rs, tests/wrap/cli_issue_329_property.rs, tests/data/issue_329_wrap_fences_ellipsis_input.txt, tests/data/issue_329_wrap_fences_ellipsis_sql_body.txt, tests/data/issue_329_wrap_fences_ellipsis_json_body.txt, tests/data/issue_329_wrap_fences_ellipsis_inner_fence.txt, tests/wrap/mod.rs	Add snapshot regression test_cli_wrap_fences_ellipsis_preserve_fenced_content and proptest combined_flags_preserve_generated_fenced_bodies. Introduce helper strategies, ISSUE_329_COMBINED_FLAGS constant, and property-test utilities to generate and locate fenced blocks. Use fixtures exercising SQL, JSON-like, and nested fenced examples and assert fenced bodies remain byte-for-byte unchanged under combined flags.

Poem

Keep the fences tight and sure,
Tracker watches, nothing tore,
Wraps that once would smear the code,
Now respect each guarded mode,
Tests proclaim the fix secure.

🚥 Pre-merge checks | ✅ 19 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
User-Facing Documentation	⚠️ Warning	PR fixes issue #329 preventing --wrap, --ellipsis, and post-wrap transforms from mutating fenced code block contents, but users-guide.md was not updated for end users.	Document in docs/users-guide.md that fenced code block bodies are preserved during --wrap and post-wrap transforms like --ellipsis, --renumber, --breaks, and --fences.

✅ Passed checks (19 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Preserve fenced code block contents during wrapping (#329)' accurately summarises the main changeset, aligning with the PR's core objective to prevent fenced code block mutation and correctly references the linked issue.
Description check	✅ Passed	The description is comprehensive and directly related to the changeset, detailing the tokeniser refactoring, test additions, validation results, and cross-referencing specific files changed.
Linked Issues check	✅ Passed	The PR successfully implements the core requirement from #329: fenced code block contents are preserved during wrapping and post-processing by refactoring tokenize_markdown to use FenceTracker, preventing --wrap and --ellipsis from mutating content between fence markers.
Out of Scope Changes check	✅ Passed	All changes are scoped to issue #329: tokeniser refactoring, shared FenceTracker integration, comprehensive test coverage (regression, snapshot, property-based, and unit tests), and documentation updates supporting the fence-preservation objective.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Testing (Overall)	✅ Passed	Unit tests verify Token::Fence vs Text discrimination; snapshot test validates full ellipsis preservation; property test with 96 cases asserts byte-exact body preservation under combined flags.
Developer Documentation	✅ Passed	Module docs in src/wrap/tokenize/mod.rs and developers-guide.md document FenceTracker integration; architecture.md references it; no multi-locale updates or roadmap changes needed.
Module-Level Documentation	✅ Passed	All five modules carry docstrings clearly explaining purpose and function. Key module src/wrap/tokenize/mod.rs documents FenceTracker integration and relationships.
Testing (Unit And Behavioural)	✅ Passed	Unit tests cover edge cases; property tests exercise 96 scenarios; snapshot test validates CLI workflow with combined flags.
Testing (Property / Proof)	✅ Passed	PR introduces fenced-block preservation invariant tested via property-based test with 96 proptest cases, plus 7 unit tests for tokeniser and snapshot test covering end-to-end behaviour.
Testing (Compile-Time / Ui)	✅ Passed	Snapshot test with insta verifies content preservation. Property-based tests (96 cases). Seven fence-tokenisation unit tests. Deterministic fixtures, no nondeterministic field redaction needed.
Unit Architecture	✅ Passed	FenceTracker explicitly mutates state via &mut self; tokenize_markdown returns immutable tokens; dependencies injected at boundaries; tests validate state transitions and integration effects.
Domain Architecture	✅ Passed	Domain invariant properly isolated: FenceTracker encapsulates fence state; Token::Fence boundary enforces preservation; post-processors respect tokens without infrastructure leakage; cleanly testable.
Observability	✅ Passed	Correctness-only refactoring: replaces local boolean with FenceTracker. No operational failures, I/O, resources. Tests and documentation adequate.
Security And Privacy	✅ Passed	No security or privacy issues detected. Test data uses safe placeholder values; no secrets, injection risks, unsafe code, or file operations introduced.
Performance And Resource Use	✅ Passed	No regression: per-line regex cost unchanged. FenceTracker state is 16 bytes. Tokens use lifetime references only. Test generators bounded at 96 cases with linear-to-linear complexity.
Concurrency And State	✅ Passed	FenceTracker uses stack-allocated, locally-scoped state with deterministic transitions; no shared state, concurrency, unsafe code, or locks; tests cover all state transitions comprehensively.
Architectural Complexity And Maintainability	✅ Passed	PR consolidates fence tracking by making tokenize_markdown use pre-existing FenceTracker, eliminating duplicate logic without introducing speculative layers, cycles, or hidden complexity.
Rust Compiler Lint Integrity	✅ Passed	No lint suppressions, excessive clones, or artificial references added. FenceTracker uses standard initialisation with mutable borrow. All exports and helpers are used.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

📋 Issue Planner

Built with CodeRabbit's Coding Plans for faster development and fewer bugs.

View plan used: #329

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch issue-329-preserve-fenced-code-block-contents-during-wrapping

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[leynos]

@coderabbitai review

[leynos]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[leynos]

@sourcery-ai review

[sourcery-ai]

Reviewer's Guide

Refactors markdown tokenization to use the shared FenceTracker so fenced code blocks (including nested literal fences) are treated as opaque during wrapping, and adds targeted property-based and snapshot tests plus documentation updates to prevent regressions for issue #329.

File-Level Changes

Change	Details	Files
Make markdown tokenizer use shared FenceTracker so all lines inside a fenced block (including nested literal fences) are emitted as Token::Fence and preserved verbatim.	Replace local boolean fence state in tokenize_markdown with super::FenceTracker::default() and use observe()/in_fence() to detect fence boundaries and interior lines. Ensure opening, interior, and closing fence lines are emitted as Token::Fence while resuming normal inline tokenization after the closing fence. Expand module-level documentation to describe shared fence tracking semantics and how they protect fenced content from post-wrap transforms like --ellipsis and --renumber.	src/wrap/tokenize/mod.rs
Strengthen CLI and tokenizer tests to cover combined flags and nuanced fenced-block behaviors, including nested and malformed fences.	Introduce ISSUE_329_COMBINED_FLAGS constant and add regression test test_cli_wrap_fences_ellipsis_preserve_fenced_content using an insta snapshot for the combined --wrap --renumber --breaks --ellipsis --fences pipeline. Add a property-based test combined_flags_preserve_generated_fenced_bodies using proptest strategies to generate various fence markers, info strings, and bodies while asserting fenced bodies are unchanged by the CLI. Add multiple new tokenize_markdown unit tests to verify closed fences, inner lines, nested shorter fences, post-fence text tokenization, and malformed fences; adjust existing tests to use slices and updated expectations. Wire the new tokenizer test module into the wrap test suite by enabling tests/wrap/tokenize_markdown.rs in tests/wrap/mod.rs.	tests/wrap/cli.rs tests/wrap/tokenize_markdown.rs tests/wrap/mod.rs tests/data/issue_329_wrap_fences_ellipsis_inner_fence.txt tests/data/issue_329_wrap_fences_ellipsis_input.txt tests/data/issue_329_wrap_fences_ellipsis_json_body.txt tests/data/issue_329_wrap_fences_ellipsis_sql_body.txt tests/snapshots/wrap__wrap_integration__cli__issue_329_wrap_fences_ellipsis_preserve_fenced_content.snap
Document the shared fence tracking behavior and its invariants in the developer guide.	Extend the developers guide with a new "Shared fence tracking" section that explains tokenize_markdown’s use of FenceTracker, its interaction with wrap_text, and guarantees about preserving fenced block contents. Record that this behavior was added for issue Preserve fenced code block contents during wrapping #329 in PR Preserve fenced code block contents during wrapping (#329) #343 and that nested literal fences shorter than the active fence are also preserved.	docs/developers-guide.md

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#329	Ensure that running mdtablefix with --wrap (and related flags such as --ellipsis, --renumber, --breaks, --fences) does not mutate the contents of fenced code blocks; all lines between opening and closing fences must be preserved byte-for-byte.	✅
#329	Handle nested and variant fenced blocks correctly (including shorter literal inner fences and different markers like ``` and ~~~) so that inner fences remain literal within an outer fence and code-like bodies (SQL, JSON, payloads, etc.) are not rewritten, while normal tokenization resumes after the closing fence.	✅
#329	Document the shared fence-tracking behavior used by tokenize_markdown so that its interaction with wrap_text and post-processing flags is clearly specified for developers.	✅

Possibly linked issues

• Preserve fenced code block contents during wrapping #329: PR updates Markdown tokenization to use FenceTracker so wrap/ellipsis don’t alter fenced code block contents, fixing Preserve fenced code block contents during wrapping #329.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[leynos]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.