Pipe escaping fix, Unicode width handling, tests, ADR 0001

[leynos]

## Summary - Implements fix for escaping pipes within the table reflow pipeline - Adds tests to ensure continuation rows preserve column boundaries and escaped pipes remain literals - Introduces internal markers for leading empty continuation cells and re-escaping logic - Updates documentation: ADR 0001 and architecture notes to reflect new pipeline behavior - Improves width handling to render by Unicode display width

Changes

Core logic

• Add parse_rows, clean_rows, calculate_widths, format_rows, insert_separator, and detect_separator to src/reflow.rs
• Implement protection and re-escaping of leading empty cells to preserve continuation boundaries
• Compute widths with UnicodeWidthStr::width to support wide Unicode chars

Table formatting

• Add format_separator_cells and related helpers in src/table.rs
• Implement extraction of separator lines and restoration of alignment markers

Tests

• Extend tests/table_continuations.rs with scenarios for:
  • Preservation of leading empty continuation cells
  • Escaped pipes in continuation rows
  • Unicode width-consistent rendering
  • Ellipsis handling in cells
• Add unit tests in reflow and table modules for new helpers:
  • protect_leading_empty_cells behavior
  • calculate_widths correctness
  • format_separator_cells alignment preservation

Documentation

• ADR 0001: Table reflow pipeline with preservation rules
• architecture.md updated with a dedicated section describing the table reflow pipeline and continuation handling
• Minor wording updates in docs/execplans/yaml-frontmatter.md to maintain consistency

Test plan

• Run cargo test
• Focused test groups:
  • cargo test --lib tests::table_continuations
  • cargo test --lib reflow
  • cargo test --lib
• Verify:
  • Escaped pipes remain literal after reparsing
  • Continuation rows preserve original column positions
  • Unicode wide characters align by display width
  • Separator cells maintain alignment markers and minimum width of three dashes

📎 Task: https://www.devboxer.com/task/eb7e6e3d-b663-4f6d-92ed-d23061ed6106

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6b403bf9-d3f9-4a88-a8e6-d94a6e13e2e4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-pipe-escaping-and-add-tests-4hb1uo

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Reviewer's Guide

Refactors the Markdown table reflow pipeline to preserve continuation-row structure and escaped pipes while making width calculations Unicode-aware, adds focused helpers and tests around separator handling, and updates architecture/ADR/docs to describe the new behavior and keep examples consistent.

Class diagram for updated reflow and table helpers

classDiagram
    class reflow_module {
        +parse_rows(trimmed_lines) Vec_Vec_String
        +clean_rows(rows) Vec_Vec_String
        +calculate_widths(rows, max_cols) Vec_usize
        +format_rows(rows, widths, indent) Vec_String
        +insert_separator(out_lines, sep_cells, indent) Vec_String
        +detect_separator(sep_line, rows, max_cols) SeparatorDetection
        -split_into_rows(cells) Vec_Vec_String
        -protect_leading_empty_cells(line) String
        -pad_cell_to_width(cell, width) String
    }

    class table_module {
        +split_cells(line) Vec_String
        +reflow_table(lines) Vec_String
        +format_separator_cells(widths, sep_cells) Vec_String
        -extract_indent_and_trim(lines) IndentAndTrimmed
        -extract_separator_line(lines) Option_String
        -rows_mismatched(rows, allow_short) bool
    }

    class process_module {
        +process_stream_inner(lines, options) Vec_String
    }

    class frontmatter_module {
        +split_frontmatter(lines) FrontmatterSplit
    }

    process_module --> frontmatter_module : uses
    process_module --> table_module : calls reflow_table

    table_module --> reflow_module : uses helpers
    reflow_module --> table_module : uses split_cells

    class UnicodeWidthStr {
        +width(text) usize
    }

    reflow_module --> UnicodeWidthStr : calculates_display_widths
    table_module --> UnicodeWidthStr : formats_separator_cells

Loading

Flow diagram for the updated table reflow pipeline

flowchart TD
    subgraph StreamProcessing
        process_stream_inner["process_stream_inner"]
        ellipsis_replacement["Apply ellipsis replacement (if enabled)"]
        reflow_table["reflow_table"]
    end

    process_stream_inner --> ellipsis_replacement
    ellipsis_replacement --> reflow_table

    subgraph TableReflowStages
        extract_indent_and_trim["extract_indent_and_trim"]
        parse_rows["parse_rows"]
        protect_leading_empty_cells["protect_leading_empty_cells"]
        clean_rows["clean_rows"]
        detect_separator["detect_separator"]
        calculate_widths["calculate_widths"]
        format_separator_cells["format_separator_cells"]
        format_rows["format_rows"]
        insert_separator["insert_separator"]
    end

    reflow_table --> extract_indent_and_trim
    extract_indent_and_trim --> parse_rows

    parse_rows --> protect_leading_empty_cells
    protect_leading_empty_cells --> parse_rows

    parse_rows --> clean_rows
    clean_rows --> detect_separator
    detect_separator --> calculate_widths

    calculate_widths --> format_separator_cells
    format_separator_cells --> format_rows
    format_rows --> insert_separator

    insert_separator --> formatted_table["Aligned table output"]

Loading

File-Level Changes

Change	Details	Files
Preserve continuation-row leading empty cells and escaped pipes during table reflow, backed by new helper functions.	Introduce parse_rows/clean_rows/calculate_widths/format_rows/insert_separator/detect_separator in the reflow pipeline to parse, clean, size, and re-emit tables. Protect leading empty continuation cells with an internal marker before splitting, then restore them after parsing so continuation columns remain aligned. Re-escape literal pipes in non-leading cells during continuation-row reconstruction so reparsing does not create extra columns. Compute column widths using UnicodeWidthStr::width and pad cells based on display width rather than byte length.	src/reflow.rs
Improve table separator formatting and add regression tests around continuation behavior, escaped pipes, Unicode widths, and ellipsis handling.	Add format_separator_cells to preserve alignment markers while enforcing a minimum separator width and respecting computed column widths. Guard separator formatting against width/column-count mismatches by returning original cells when inputs are inconsistent. Add rstest-based tests to verify continuation-row preservation, literal ellipsis behavior under different options, escaped-pipe handling in continuation rows, and Unicode-width-consistent alignment. Add unit tests for protect_leading_empty_cells, calculate_widths, and format_separator_cells to pin the new behavior.	src/table.rs tests/table_continuations.rs
Document the table reflow pipeline, its preservation rules, and refine existing documentation formatting.	Add ADR 0001 describing the table reflow pipeline, continuation-row protection, escaped-pipe handling, Unicode width rules, and ellipsis ordering. Extend architecture.md with a dedicated section on the table reflow pipeline, including stage-by-stage behavior and rationale. Tidy YAML frontmatter ADR prose for line-wrapping consistency without changing semantics. Normalize tables, code fences, and minor prose in Rust testing and documentation guides for consistency with the new style.	docs/adrs/0001-table-reflow-pipeline.md docs/architecture.md docs/execplans/yaml-frontmatter.md docs/rust-testing-with-rstest-fixtures.md docs/rust-doctest-dry-guide.md docs/documentation-style-guide.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34f89a5768

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve escaped pipes in emitted continuation cells

Re-escaping at protect_leading_empty_cells is only temporary here: split_cells unescapes \| back to | before formatting, so continuation rows with escaped pipes are emitted with a raw pipe character. In inputs like | | keep \| literal in continuation |, the output row becomes | ... | keep | literal ... |, which Markdown parsers read as an extra column delimiter and therefore still corrupts table structure. The fix needs to keep these pipes escaped in final output (or re-escape during row formatting), not only during intermediate parsing.

Useful? React with 👍 / 👎.

[leynos]

@coderabbitai Have the following now been resolved?

Annul any requirements that violate the en-GB-oxendict spelling (-ize / -yse / -our) conventions (for example a request to replace "normalize" with "normalise" or "artefact" with "artifact"), or where the requirement unnecessarily increases cyclomatic complexity.

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="tests/table_continuations.rs" line_range="70-66" />
<code_context>
 }

+#[test]
+fn process_stream_preserves_literal_ellipsis_in_table_cells_when_disabled() {
+    let input = lines_vec![
+        "| Module | Notes                       |",
+        "| ------ | --------------------------- |",
+        "| `foo`  | preserves literal ... here  |",
+        "| `bar`  | and also ... in this cell   |",
+    ];
+    let expected = lines_vec![
+        "| Module | Notes                      |",
+        "| ------ | -------------------------- |",
+        "| `foo`  | preserves literal ... here |",
+        "| `bar`  | and also ... in this cell  |",
+    ];
+
+    assert_eq!(process_stream(&input), expected);
+}
+
</code_context>
<issue_to_address>
**issue (testing):** Test name implies explicit ellipsis disabling, but the test relies on the default configuration

This test uses `process_stream` with default options while its name says "when_disabled". If the default changes, the test will no longer cover the "disabled" behavior. Please either pass an explicit options struct via `process_stream_opts` that disables ellipsis replacement (preferred), or rename the test to indicate it verifies the default behavior.
</issue_to_address>

### Comment 2
<location path="docs/architecture.md" line_range="125" />
<code_context>
+3. `clean_rows`, `detect_separator`, and `calculate_widths` rebuild the logical
+   table. Explicit separator lines are preferred, but the second parsed row can
+   be promoted when the source embeds the separator in the body. Widths are
+   measured with `UnicodeWidthStr::width`, so CJK text, emoji, and accented
+   characters align by display width rather than byte count.
+4. `format_rows` and `insert_separator` emit the final table. Separator cells
</code_context>
<issue_to_address>
**suggestion (review_instructions):** The acronym “CJK” is introduced without being expanded on first use.

Consider expanding the acronym on first use, for example: “Chinese, Japanese, and Korean (CJK) text, emoji, and accented …”.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.md`

**Instructions:**
Define uncommon acronyms on first use.

</details>
</issue_to_address>

### Comment 3
<location path="docs/architecture.md" line_range="142" />
<code_context>
+aligning for `...` and shrinking the rendered column after the fact.
+
+The rationale for these choices is captured in
+[ADR 0001](adrs/0001-table-reflow-pipeline.md).
+
 ## Footnote Conversion
</code_context>
<issue_to_address>
**suggestion (review_instructions):** The acronym “ADR” is used in link text without being expanded on first use in this document segment.

To satisfy the acronym rule, consider something like “Architecture Decision Record (ADR) 0001” for the first mention.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.md`

**Instructions:**
Define uncommon acronyms on first use.

</details>
</issue_to_address>

### Comment 4
<location path="docs/adrs/0001-table-reflow-pipeline.md" line_range="1" />
<code_context>
+# ADR 0001: preserve table structure during reflow
+
+- Status: Accepted
</code_context>
<issue_to_address>
**suggestion (review_instructions):** The document title uses the acronym “ADR” without defining it on first use.

Consider updating the title to expand the acronym, for example: “Architecture Decision Record (ADR) 0001: preserve table structure during reflow”.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.md`

**Instructions:**
Define uncommon acronyms on first use.

</details>
</issue_to_address>

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check Comment 1: test name / usage in table_continuations.rs
rg -n "process_stream_preserves_literal_ellipsis" tests/table_continuations.rs -A 5

# Check Comments 2 & 3: CJK and ADR in architecture.md
grep -n "CJK\|ADR\|Chinese" docs/architecture.md

# Check Comment 4: title line of ADR document
head -5 docs/adrs/0001-table-reflow-pipeline.md

Length of output: 696

---

@leynos: Comments 2, 3, and 4 are resolved. Comment 1 is not yet addressed.

#	Location	Status
1	tests/table_continuations.rs line 70	❌ Not resolved
2	docs/architecture.md line 125 — "CJK" expansion	✅ Resolved
3	docs/architecture.md line 143 — "ADR" expansion in link text	✅ Resolved
4	docs/adrs/0001-table-reflow-pipeline.md title	✅ Resolved

Comment 1 still requires attention. The function at line 70 retains the name process_stream_preserves_literal_ellipsis_in_table_cells_when_disabled and calls process_stream with the default configuration. Either rename it to reflect default behaviour, or switch to process_stream_opts with an explicit struct that disables ellipsis replacement. Example of the latter:

#[test]
fn process_stream_preserves_literal_ellipsis_in_table_cells_when_disabled() {
    let opts = Opts {
        ellipsis: false,
        ..Opts::default()
    };
    let input = lines_vec![
        "| Module | Notes                       |",
        "| ------ | --------------------------- |",
        "| `foo`  | preserves literal ... here  |",
        "| `bar`  | and also ... in this cell   |",
    ];
    let expected = lines_vec![
        "| Module | Notes                      |",
        "| ------ | -------------------------- |",
        "| `foo`  | preserves literal ... here |",
        "| `bar`  | and also ... in this cell  |",
    ];

    assert_eq!(process_stream_opts(&input, &opts), expected);
}

Adjust the Opts field name to match the actual struct definition in the codebase.