fix: escape pipes and newlines in CSV values

[Asjad Abbas (asjad3)]

Problem

CsvConverter builds its Markdown table with " | ".join(...) and passes cell values
through unescaped. Two characters that are perfectly legal inside a CSV field corrupt
the table, and neither raises:

• a pipe is read as a column separator, so the row gains columns the header never
  declared
• a newline inside a quoted field ends the row early, leaving the rest as stray text
  after the table

The pipe case loses data rather than just looking wrong. Given cheap | fast:

name,description
Widget,"cheap | fast"

main emits a row with three columns under a two-column header:

| name | description |
| --- | --- |
| Widget | cheap | fast |

A renderer drops the surplus column, so the word fast disappears entirely:

<tr><td>Widget</td><td>cheap</td></tr>   <!-- "fast" is gone -->

With this change the pipe is escaped and the value survives:

<tr><td>Widget</td><td>cheap | fast</td></tr>

(HTML above is python-markdown with the tables extension, before and after.)

Change

A small _escape_table_cell() helper applied to header and data cells: pipes are
escaped as \|, and CR/LF/CRLF inside a quoted field become <br> so the record stays
on one line.

Deliberately narrow:

• Backslashes are left alone. Escaping them would be more theoretically correct but
  changes output for ordinary content like Windows paths, with no reported problem to
  justify it.
• <br> is the usual convention for a line break inside a GFM table cell; the
  alternative is collapsing to a space, which loses the author's intent.

Scope note

The HTML converter has the same gap — <td>cheap | fast</td> also renders as a split
cell — and since XlsxConverter routes through DataFrame.to_html() → HtmlConverter,
it inherits it. I left that alone here: it runs through markdownify rather than a
hand-rolled join, so it is a different fix with much wider blast radius across existing
outputs. Happy to follow up if you would like it handled, and equally happy to fold it
in here if you would rather have one change.

Tests

packages/markitdown/tests/test_csv_escaping.py — four cases: pipe in a cell, pipe in
the header, newline in a quoted cell, and a guard that ordinary content is not
over-escaped. The first three fail on main and pass with this change; the fourth
passes either way and exists to catch over-escaping.

tests/test_csv_escaping.py ....                    [100%]
4 passed

test_module_vectors.py + test_module_misc.py: 121 passed, 2 skipped, 1 failed —
test_speech_transcription, which fails identically on a clean checkout here (audio
model output, unrelated to this change). The existing test_mskanji.csv vector contains
no pipes or embedded newlines and is unaffected.

black clean.

---

Written with AI assistance (Claude Code); I reviewed the change and ran the checks
above.

[Asjad Abbas (asjad3)]

@microsoft-github-policy-service agree

[Copilot]

🟡 Changes recommended

Newline handling fails its regression test, and backslash-prefixed pipes can still split cells.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds CSV cell escaping to prevent Markdown table corruption from pipes and embedded newlines.

Changes:

• Escapes pipes in headers and data cells.
• Adds regression tests for pipe and newline handling.

File summaries

File	Description
packages/markitdown/src/markitdown/converters/_csv_converter.py	Adds cell escaping during table generation.
packages/markitdown/tests/test_csv_escaping.py	Tests CSV escaping behavior.

Review details

Suppressed comments (1)

packages/markitdown/src/markitdown/converters/_csv_converter.py:21

• These replacements contradict the documented behavior and make test_newline_in_quoted_cell_does_not_split_the_row fail: the converter emits line one line two, while line 47 expects line one<br>line two. Replacing line breaks with spaces also loses the author’s intended break, so use <br> for each newline form as described by the PR.

        .replace("\r\n", " ")
        .replace("\n", " ")
        .replace("\r", " ")

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

The implemented newline behavior contradicts the PR’s stated <br> contract.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

[afourney]

Intentional.