--ellipsis corrupts literal '...' inside indented (4-space) code blocks

[lodyai]

Summary

mdtablefix --ellipsis rewrites literal ... to the Unicode ellipsis …
inside indented (four-space) code blocks. Indented code blocks are not
protected the way fenced blocks and inline code are, so literal example
content (command output, source snippets) is silently corrupted.

Version

mdtablefix 0.4.0

Reproduction

Input (ellipsis.md):

# Ellipsis repro

Expected test output:

    running 2 tests
    test foo ... ok
    test bar ... ok
    ...
    test result: ok

Inline code is fine: `foo(...)` stays literal.

```text
fenced ... stays literal
```

Command:

mdtablefix --wrap --renumber --breaks --ellipsis --fences --in-place ellipsis.md

Actual behaviour

The indented code block is rewritten (inline code and the fenced block are
correctly left alone):

<     test foo ... ok
<     test bar ... ok
<     ...
---
>     test foo … ok
>     test bar … ok
>     …

Expected behaviour

Indented code blocks should be treated as literal, exactly like fenced code
blocks and inline code spans. ... inside an indented code block must not be
converted to ….

Impact

This is a silent semantic break: the corrupted content is real code / command
output, and because the resulting lines are usually under 80 columns,
markdownlint does not flag it. Real-world hits were sample cargo test output
(test ... ok) and a Rust snippet whose string literals "..." became "…".

[coderabbitai]

🔗 Related PRs

#280 - Keep GFM footnote references atomic (#277) [merged]
#343 - Preserve fenced code block contents during wrapping (#329) [merged]
#344 - Preserve inline code spans when wrapping prose (#326) (#323) (#327) (#310) [merged]
#351 - Fix backslash-ended code span wrapping (#349) [merged]
#356 - Keep escaped backtick code spans atomic (#354) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[leynos]

@CodeRabbit plan

[coderabbitai]

Coding Plan

Summary

Fix the root cause in the shared tokenizer rather than patching ellipsis alone, so code_emphasis and footnotes inherit correct indented-code protection.

Add a stateful indented-code classifier modelled on FenceTracker, reusing leading_indent, and honour the "cannot interrupt a paragraph" rule via a preceding-blank requirement.

Represent indented-code content with a new Token::IndentedCode variant forwarded verbatim, updating only the few exhaustive token matches.

Guard the behaviour with unit, tokenizer, and end-to-end tests, and record the fix in the changelog and architecture/developer docs.

Design Choices

Design Choice 1: How should indented-code protection be introduced into the token stream?

Options Considered:

1. Add a dedicated Token::IndentedCode variant emitted by tokenize_markdown, forwarded verbatim by all consumers.
2. Re-use the existing opaque Token::Fence variant for indented code lines (zero consumer changes, since Fence is already treated as opaque passthrough), at the cost of conflating two distinct constructs.
3. Pre-filter indented lines inside ellipsis.rs only, before process_tokens.

Chosen Option: 1

Rationale: An explicit variant is the clearest, self-documenting choice, centralises the fix so code_emphasis and footnotes inherit it correctly, and only requires updating a small number of exhaustive matches; Option 3 duplicates logic and leaves sibling transforms buggy, and Option 2 sacrifices clarity for a marginal saving.

Design Choice 2: How faithfully should CommonMark blank-line and paragraph-interruption rules be modelled?

Options Considered:

1. Full CommonMark lazy blank-line lookahead (buffer trailing blanks, include only if resumed).
2. A simplified stateful rule: an indented-content line is code if the previous line was blank (or the document/block start) or a code block is already open.

Chosen Option: 2

Rationale: Because blank lines are emitted identically regardless of classification, the simplified rule produces correct output while still honouring the "cannot interrupt a paragraph" rule via the preceding-blank requirement, and avoids unnecessary lookahead complexity.

Design Choice 3: Scope of list-context-relative indented code detection

Options Considered:

1. Model list-context-relative indented code (four spaces beyond a list item's content column).
2. Treat detection as absolute-column, matching the existing leading_indent/is_indented_code_line behaviour.

Chosen Option: 2

Rationale: List-context-relative indented code is out of scope. No existing module models this, and the ticket's cases are top-level indented blocks. Treat detection as absolute-column, matching the existing leading_indent/is_indented_code_line behaviour.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Teach the shared tokenizer to protect indented code blocks

Give the shared tokenizer first-class awareness of four-space indented code blocks so that every token-based transform receives their content as opaque, verbatim tokens. This is the heart of the fix: introduce a small stateful classifier mirroring the existing fence tracker, emit a distinct token for indented-code content, and update the handful of consumers that match the token enum exhaustively so they forward the new token unchanged.

Task 1: Add a stateful indented-code-block classifier

Introduce a tracker analogous to FenceTracker that decides, line by line, whether the current line belongs to an indented code block.

• Add a small stateful type (e.g. IndentedCodeTracker) alongside FenceTracker in src/wrap/fence.rs, or in a sibling module under src/wrap/tokenize/, following the established (state, observe() -> bool, in_block() -> bool) shape.
• Reuse leading_indent from src/wrap/block.rs (already pub(crate)) to compute the tab-aware indentation width; treat a width >= 4 with non-whitespace content as an indented-code candidate.
• Track two pieces of state: whether an indented code block is currently open, and whether the previous line was blank (with document/block start counting as "blank" so a leading indented block opens correctly).
• Classify a candidate line as indented code when a block is already open or the previous line was blank; this honours the CommonMark rule that an indented code block cannot interrupt a paragraph.
• Close the block on any non-blank line whose indentation width is < 4; keep it open across interior blank lines.
• Fence lines take precedence: ensure the tracker's state does not open an indented block from content that the fence checks have already consumed.

Task 2: Emit indented-code content as a dedicated token

Wire the new classifier into the tokenizer's per-line loop and represent indented-code content with its own token variant.

• Add a Token::IndentedCode(&'a str) variant to the Token enum in src/wrap/tokenize/mod.rs.
• In tokenize_markdown's per-line loop, add a third early branch after the two existing fence checks (observe and in_fence) and before tokenize_inline: when the classifier reports the line as indented code, push Token::IndentedCode(line) verbatim, emit the trailing newline as usual, and continue.
• Keep fence handling first so that fenced-block interiors at any indentation remain fence content and are never reinterpreted as indented code.

Task 3: Forward the new token verbatim in exhaustive consumers

Update the token consumers that match the enum without a wildcard so the code compiles and the new content is passed through unchanged.

• Update push_original_token in src/textproc.rs to render Token::IndentedCode(s) by pushing its raw string, exactly like Token::Fence.
• Update the exhaustive match in fix_code_emphasis (src/code_emphasis.rs) to forward Token::IndentedCode verbatim, so emphasis/backtick markers inside indented code are left literal.
• Update the exhaustive matches in src/textproc.rs's doctest and unit tests to cover the new variant.
• Confirm the wildcard-based consumers (src/ellipsis.rs, src/footnotes/mod.rs, src/footnotes/renumber.rs) require no code changes: their catch-all arms already forward the new variant to push_original_token, which is the correct pass-through behaviour.

🤖 Prompt for AI agents

Implement Phase 1: teach the shared Markdown tokenizer to treat four-space
indented code blocks as opaque, verbatim content, so every downstream
token-based transform (ellipsis, code_emphasis, footnotes) leaves them
untouched.

- Add a stateful `IndentedCodeTracker` classifier, modelled on the existing
`FenceTracker` in `src/wrap/fence.rs` (same `(state, observe() -> bool,
in_block() -> bool)` shape), either alongside `FenceTracker` in that file or in
a sibling module under `src/wrap/tokenize/`.
- Reuse `leading_indent` from `src/wrap/block.rs` to compute tab-aware
indentation width. A candidate indented-code line has width `>= 4` with
non-whitespace content.
- Track whether an indented code block is currently open and whether the
previous line was blank (treat document/block start as blank). Classify a
candidate line as indented code when a block is already open OR the previous
line was blank — this enforces the CommonMark rule that an indented code block
cannot interrupt a paragraph.
- Close the block on any non-blank line with indentation width `< 4`; keep the
block open across interior blank lines.
- Ensure fence checks take precedence over the indented-code tracker so fenced
content is never misclassified.
- Add a `Token::IndentedCode(&'a str)` variant to the `Token` enum in
`src/wrap/tokenize/mod.rs`. In `tokenize_markdown`'s per-line loop, insert a
third early branch after the two existing fence checks (`observe` and
`in_fence`) and before `tokenize_inline`: when the classifier flags the line as
indented code, push `Token::IndentedCode(line)` verbatim, emit the trailing
newline, and `continue`.
- Update every exhaustive match over `Token` so the new variant compiles and
forwards content unchanged: `push_original_token` in `src/textproc.rs` (push the
raw string, exactly like `Token::Fence`), the exhaustive match in
`fix_code_emphasis` in `src/code_emphasis.rs`, and the doctest/unit-test matches
in `src/textproc.rs`.
- Leave the wildcard-based consumers (`src/ellipsis.rs`, `src/footnotes/mod.rs`,
`src/footnotes/renumber.rs`) unchanged — confirm their catch-all arms already
forward the new variant correctly via `push_original_token`.

Phase 2: Prove the fix and guard against regressions

Add targeted unit tests at the tokenizer and transform level, plus an end-to-end test mirroring the ticket's reproduction, establishing the "pass-through untouched" expectation that is currently missing for indented code blocks.

Task 1: Add ellipsis unit coverage

Assert that replace_ellipsis leaves indented code blocks literal.

• Extend the inline #[cfg(test)] mod tests in src/ellipsis.rs with an ignores_indented_code_blocks case following the existing ignores_fenced_blocks idiom (Vec<String> input/expected).
• Include the ticket's key cases: an indented ... line, an indented test foo ... ok line, and a boundary check that a three-space-indented ... (below the threshold) is still converted.
• Cover a preceding-blank-line boundary and an interior blank line within the indented block.

Task 2: Add tokenizer and sibling-transform coverage

Lock in the tokenizer behaviour and confirm the sibling transforms inherit protection.

• Add tokenizer tests (in the tokenize test module and/or tests/wrap/) asserting that indented-code lines emit Token::IndentedCode, that fences still take precedence, and that a line following a paragraph without an intervening blank is not treated as indented code.
• Follow the established rstest #[case(...)] boundary-table convention seen in src/wrap/block.rs and src/process/buffer/tests.rs for indentation thresholds.
• Add a code_emphasis pass-through test confirming backtick/emphasis markers inside an indented block are left literal.

Task 3: Add an end-to-end regression test

Reproduce the ticket scenario through the CLI/processing pipeline.

• Add an integration test that runs the combined transform flow (matching the ticket's --ellipsis usage) over the reproduction document and asserts the indented block's ... stays literal while the inline foo(...) and the fenced block remain correctly untouched.

🤖 Prompt for AI agents

Implement Phase 2: add tests that prove indented code blocks are protected by
the tokenizer fix and guard against regressions.

- In `src/ellipsis.rs`'s inline `#[cfg(test)] mod tests`, add an
`ignores_indented_code_blocks` test following the existing
`ignores_fenced_blocks` idiom (`Vec<String>` input/expected). Cover: an indented
`...` line, an indented `test foo ... ok` line, a boundary check that a
three-space-indented `...` (below the four-space threshold) IS still converted,
a preceding-blank-line boundary case, and an interior blank line within the
indented block.
- Add tokenizer tests (in the tokenize test module and/or `tests/wrap/`)
asserting `Token::IndentedCode` is emitted for indented-code lines, that fences
take precedence over indented-code classification, and that a line following a
paragraph without an intervening blank line is NOT treated as indented code. Use
the established `rstest` `#[case(...)]` boundary-table convention seen in
`src/wrap/block.rs` and `src/process/buffer/tests.rs` for indentation
thresholds.
- Add a `code_emphasis` pass-through test confirming backtick/emphasis markers
inside an indented code block remain literal.
- Add an integration test reproducing the ticket's `--ellipsis` CLI/pipeline
scenario, asserting the indented block's `...` stays literal while the inline
`foo(...)` and the fenced block remain correctly untouched.

Phase 3: Update documentation and changelog

Record the corrected behaviour in line with repository conventions so the shared-tokenizer invariant is discoverable.

Task 1: Add a changelog entry

Document the fix under the Unreleased section.

• Add a bullet under ### Fixed in ## [Unreleased] in CHANGELOG.md, phrased as the corrected behaviour with a trailing issue link, matching the existing "Preserve fenced and indented code blocks verbatim..." wording style. Example intent: "Preserve four-space indented code blocks verbatim when --ellipsis is used, so literal ... in command output and code snippets is not converted to …."

Task 2: Update architecture and developer documentation

Reflect the tokenizer's new invariant.

• Update the tokenizer prose in docs/architecture.md (the "Markdown stream processor" / "Tokenizer flow" area) to note that tokenize_markdown now emits indented-code content as verbatim tokens alongside fenced content, naming the new classifier and Token::IndentedCode.
• Add a "Design constraints" bullet to docs/developers-guide.md (mirroring the existing "Shared fence tracking" bullet) citing the issue/PR, recording that indented-code protection is now shared across all token consumers.
• No docs/contents.md change is required unless a new document is added; edit existing documents in place.

🤖 Prompt for AI agents

Implement Phase 3: document the fix in the changelog and architecture/developer
guides.

- Add a bullet under `### Fixed` in `## [Unreleased]` in `CHANGELOG.md`, phrased
as the corrected behaviour with a trailing issue link, matching the existing
"Preserve fenced and indented code blocks verbatim..." wording style. Example
intent: "Preserve four-space indented code blocks verbatim when `--ellipsis` is
used, so literal `...` in command output and code snippets is not converted to
`…`."
- Update the tokenizer prose in `docs/architecture.md` (the "Markdown stream
processor" / "Tokenizer flow" area) to note that `tokenize_markdown` now emits
indented-code content as verbatim tokens alongside fenced content, naming the
new classifier and `Token::IndentedCode`.
- Add a "Design constraints" bullet to `docs/developers-guide.md`, mirroring the
existing "Shared fence tracking" bullet, citing the issue/PR and recording that
indented-code protection is now shared across all token consumers.
- Do not touch `docs/contents.md` unless a new document is added; edit existing
documents in place.

Research

Text post-processors (ellipsis, code_emphasis, footnotes) all funnel through a shared tokenizer, wrap::tokenize_markdown (src/wrap/tokenize/mod.rs), exposed via src/textproc.rs (process_tokens, push_original_token). That tokenizer walks the source line by line, consulting a stateful FenceTracker (src/wrap/fence.rs) to emit fenced content as opaque Token::Fence and inline spans as Token::Code, while everything else becomes Token::Text. Only Token::Text is transformed by replace_ellipsis. Indented-code detection exists (is_indented_code_line in src/wrap.rs, built on leading_indent in src/wrap/block.rs), but it lives in the wrapping pass and never reaches the tokenizer, so indented code lines are emitted as ordinary Token::Text and are exposed to the ellipsis substitution.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: teach the shared Markdown tokenizer to treat four-space
indented code blocks as opaque, verbatim content, so every downstream
token-based transform (ellipsis, code_emphasis, footnotes) leaves them
untouched.

- Add a stateful `IndentedCodeTracker` classifier, modelled on the existing
`FenceTracker` in `src/wrap/fence.rs` (same `(state, observe() -> bool,
in_block() -> bool)` shape), either alongside `FenceTracker` in that file or in
a sibling module under `src/wrap/tokenize/`.
- Reuse `leading_indent` from `src/wrap/block.rs` to compute tab-aware
indentation width. A candidate indented-code line has width `>= 4` with
non-whitespace content.
- Track whether an indented code block is currently open and whether the
previous line was blank (treat document/block start as blank). Classify a
candidate line as indented code when a block is already open OR the previous
line was blank — this enforces the CommonMark rule that an indented code block
cannot interrupt a paragraph.
- Close the block on any non-blank line with indentation width `< 4`; keep the
block open across interior blank lines.
- Ensure fence checks take precedence over the indented-code tracker so fenced
content is never misclassified.
- Add a `Token::IndentedCode(&'a str)` variant to the `Token` enum in
`src/wrap/tokenize/mod.rs`. In `tokenize_markdown`'s per-line loop, insert a
third early branch after the two existing fence checks (`observe` and
`in_fence`) and before `tokenize_inline`: when the classifier flags the line as
indented code, push `Token::IndentedCode(line)` verbatim, emit the trailing
newline, and `continue`.
- Update every exhaustive match over `Token` so the new variant compiles and
forwards content unchanged: `push_original_token` in `src/textproc.rs` (push the
raw string, exactly like `Token::Fence`), the exhaustive match in
`fix_code_emphasis` in `src/code_emphasis.rs`, and the doctest/unit-test matches
in `src/textproc.rs`.
- Leave the wildcard-based consumers (`src/ellipsis.rs`, `src/footnotes/mod.rs`,
`src/footnotes/renumber.rs`) unchanged — confirm their catch-all arms already
forward the new variant correctly via `push_original_token`.
===============================================================================

Task: 2

Implement Phase 2: add tests that prove indented code blocks are protected by
the tokenizer fix and guard against regressions.

- In `src/ellipsis.rs`'s inline `#[cfg(test)] mod tests`, add an
`ignores_indented_code_blocks` test following the existing
`ignores_fenced_blocks` idiom (`Vec<String>` input/expected). Cover: an indented
`...` line, an indented `test foo ... ok` line, a boundary check that a
three-space-indented `...` (below the four-space threshold) IS still converted,
a preceding-blank-line boundary case, and an interior blank line within the
indented block.
- Add tokenizer tests (in the tokenize test module and/or `tests/wrap/`)
asserting `Token::IndentedCode` is emitted for indented-code lines, that fences
take precedence over indented-code classification, and that a line following a
paragraph without an intervening blank line is NOT treated as indented code. Use
the established `rstest` `#[case(...)]` boundary-table convention seen in
`src/wrap/block.rs` and `src/process/buffer/tests.rs` for indentation
thresholds.
- Add a `code_emphasis` pass-through test confirming backtick/emphasis markers
inside an indented code block remain literal.
- Add an integration test reproducing the ticket's `--ellipsis` CLI/pipeline
scenario, asserting the indented block's `...` stays literal while the inline
`foo(...)` and the fenced block remain correctly untouched.
===============================================================================

Task: 3

Implement Phase 3: document the fix in the changelog and architecture/developer
guides.

- Add a bullet under `### Fixed` in `## [Unreleased]` in `CHANGELOG.md`, phrased
as the corrected behaviour with a trailing issue link, matching the existing
"Preserve fenced and indented code blocks verbatim..." wording style. Example
intent: "Preserve four-space indented code blocks verbatim when `--ellipsis` is
used, so literal `...` in command output and code snippets is not converted to
`…`."
- Update the tokenizer prose in `docs/architecture.md` (the "Markdown stream
processor" / "Tokenizer flow" area) to note that `tokenize_markdown` now emits
indented-code content as verbatim tokens alongside fenced content, naming the
new classifier and `Token::IndentedCode`.
- Add a "Design constraints" bullet to `docs/developers-guide.md`, mirroring the
existing "Shared fence tracking" bullet, citing the issue/PR and recording that
indented-code protection is now shared across all token consumers.
- Do not touch `docs/contents.md` unless a new document is added; edit existing
documents in place.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!