HTML And Markdown Cleaning

HTML And Markdown Cleaning

ContextClean is designed to turn raw web exports into readable model context.

What It Removes

The HTML cleaner removes high-confidence noise such as:

• script
• style
• svg
• comments
• navigation blocks
• footers
• asides
• cookie banners
• newsletter modals
• ad blocks
• tracking blocks

The aggressive mode removes more boilerplate, but the cleaner is still designed to preserve visible main content.

What It Preserves

The cleaner is intended to preserve:

• headings
• paragraphs
• links
• lists
• tables
• inline code
• fenced code blocks
• readable text order
• useful article or documentation structure

Example

ctxclean fixtures/dirty_html_article.html --mode standard --format markdown

Expected behavior:

• tracking scripts are removed
• cookie prompts are removed
• the main article remains readable
• code blocks stay code-shaped
• links remain visible in Markdown output

Malformed HTML

Modern web exports are often malformed. ContextClean uses parser-backed cleanup for malformed nested structures and falls back conservatively when needed.

Regression coverage includes malformed browser-export fixtures with:

• nested lists
• links
• inline code
• tables
• code blocks
• boilerplate that should be removed

Markdown Input

Markdown is cleaned as text with additional handling for repeated noise, boilerplate, logs, and comments depending on options. Fenced code block boundaries are treated carefully because splitting or corrupting code blocks can make model context worse.

Practical Advice

• Use standard first for documentation and articles.
• Use aggressive for pages dominated by boilerplate.
• Use json when you need to inspect removed sections and warnings.
• Use report when you want to explain why the input is noisy.