feat: markdown-aware TTS text normalization#1
Conversation
Add handle_markdown() pre-processing pass that strips markdown formatting (headings, bold, italic, links, code blocks, tables, etc.) before TTS synthesis. Preserves code block contents literally and avoids false positives on snake_case identifiers. - New: handle_markdown() function in normalizer.py - New: markdown_normalization toggle in NormalizationOptions (default True) - New: test_normalizer.py with 24 test cases including mixed LLM output - Fix: trailing whitespace in normalize_text() output (.strip()) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Summary of ChangesHello @Fieldnote-Echo, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the Text-to-Speech (TTS) system by introducing a robust mechanism to preprocess text containing Markdown. The primary goal is to improve the clarity and naturalness of spoken output by systematically removing visual formatting syntax, such as bolding, links, and code blocks, before the text is synthesized. This ensures that the TTS engine focuses solely on articulating the textual content, leading to a more pleasant listening experience. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces markdown-aware text normalization for TTS synthesis, which is a valuable addition for handling varied input formats. The handle_markdown function is well-designed, employing a clever placeholder strategy to protect code blocks from premature stripping. The integration into normalize_text is correctly prioritized to ensure markdown links are processed before URL normalization. The new test suite is comprehensive, covering a wide range of markdown elements and edge cases, which significantly boosts confidence in the new functionality.
| _MD_INLINE_CODE = re.compile(r"`([^`]+)`") | ||
| _MD_IMAGE = re.compile(r"!\[([^\]]*)\]\([^)]+\)") | ||
| _MD_LINK = re.compile(r"\[([^\]]+)\]\([^)]+\)") | ||
| _MD_BOLD = re.compile(r"\*\*(.+?)\*\*") |
There was a problem hiding this comment.
The current _MD_BOLD regex only handles **bold**. Markdown also supports __bold__ for bold text. To ensure comprehensive markdown stripping, consider updating this regex to include __ as well, using a backreference to match the opening and closing delimiters.
| _MD_BOLD = re.compile(r"\*\*(.+?)\*\*") | |
| _MD_BOLD = re.compile(r"(\*\*|__)(.+?)\1") |
| # Table pipes — only replace on lines that look like table rows (2+ pipes). | ||
| # This preserves non-table pipes like "true | false". | ||
| lines = text.split("\n") | ||
| for i, line in enumerate(lines): | ||
| if line.count("|") >= 2: | ||
| lines[i] = line.replace("|", " ") | ||
| text = "\n".join(lines) |
There was a problem hiding this comment.
The current approach to stripping table pipes relies on a heuristic (line.count("|") >= 2). While this works for typical markdown tables, it might inadvertently affect non-table lines that happen to contain two or more pipe characters. For example, a sentence like "This is option A | option B | option C" would have its pipes replaced, which might not be the desired TTS output. Consider refining this logic to be more specific to actual table structures, perhaps by checking for header/separator rows more explicitly, or by using a more robust markdown parsing library if this becomes a frequent issue.
1 participant
Summary
handle_markdown()pre-processing pass that strips markdown formatting before TTS synthesismarkdown_normalizationtoggle inNormalizationOptions(defaultTrue)Changes
api/src/services/text_processing/normalizer.py— newhandle_markdown()function + integration intonormalize_text()api/src/structures/schemas.py— newmarkdown_normalizationfield onNormalizationOptionsapi/src/services/text_processing/test_normalizer.py— standalone test suite (no Kokoro deps required)Test plan
pytest api/src/services/text_processing/test_normalizer.py -vpasses all 24 tests/v1/audio/speech🤖 Generated with Claude Code