fix!: syncHtmlToMarkdown -> htmlToMarkdown

Author: harlan-zw

CLAUDE.md

@@ -36,8 +36,10 @@ When you finish a task, always run `pnpm typecheck` to ensure that the code is t
 - Test all: `pnpm test`
 - Test single file: `pnpm test path/to/test.ts`
 - Test with pattern: `pnpm test -t "test pattern"`
-- Test GitHub markdown: `pnpm test:github`
-- Development: `pnpm dev:prepare`
+- Test folder: `pnpm test test/unit/plugins/`
+- Development build (stub): `pnpm dev:prepare`
+- Live test with real sites: `pnpm test:github:live`, `pnpm test:wiki:file`
+- Benchmarking: `pnpm bench:stream`, `pnpm bench:string`
 
 ## Code Style Guidelines
 - Indentation: 2 spaces
@@ -52,11 +54,25 @@ When you finish a task, always run `pnpm typecheck` to ensure that the code is t
 - Follow ESLint config based on @antfu/eslint-config
 
 ## Project Architecture
-- Core modules:
-  - `parser.ts`: Handles HTML parsing into a DOM-like structure
-  - `markdown.ts`: Transforms DOM nodes to Markdown
-  - `htmlStreamAdapter.ts`: Manages HTML streaming conversion
-  - `index.ts`: Main entry point with primary API functions
+
+### Core Architecture
+- `src/index.ts`: Main entry point with `syncHtmlToMarkdown` and `streamHtmlToMarkdown` APIs
+- `src/parser.ts`: Manual HTML parsing into DOM-like structure for performance
+- `src/markdown.ts`: DOM node to Markdown transformation logic
+- `src/stream.ts`: Streaming HTML processing with content-based buffering
+- `src/types.ts`: Core TypeScript interfaces for nodes, plugins, and state management
+
+### Plugin System
+- `src/pluggable/plugin.ts`: Plugin creation utilities and base interfaces
+- `src/plugins/`: Built-in plugins (filter, extraction, tailwind, readability, etc.)
+- `src/libs/query-selector.ts`: CSS selector parsing logic shared across plugins
+- Plugin hooks: `beforeNodeProcess`, `onNodeEnter`, `onNodeExit`, `processTextNode`
+
+### Key Concepts
+- **Node Types**: ElementNode (HTML elements) and TextNode (text content) with parent/child relationships
+- **Streaming Architecture**: Processes HTML incrementally using buffer regions and optimal chunk boundaries
+- **Plugin Pipeline**: Each plugin can intercept and transform content at different processing stages
+- **Memory Efficiency**: Immediate processing and callback patterns to avoid collecting large data structures
 
 ## Technical Details
 - Parser: Manual HTML parsing for performance, doesn't use browser DOM
@@ -75,12 +91,25 @@ When you finish a task, always run `pnpm typecheck` to ensure that the code is t
   - `--chunk-size <size>`: Controls stream chunking (default: 4096)
   - `-v, --verbose`: Enables debug logging
 
-Always run tests after making changes to ensure backward compatibility.
+## CLI and Testing
+
+### CLI Usage
+- Processes HTML from stdin, outputs Markdown to stdout
+- Test with live sites: `curl -s https://example.com | node ./bin/mdream.mjs --origin https://example.com`
+- Key CLI options: `--origin <url>`, `-v/--verbose`, `--chunk-size <size>`
 
-## Docs
+### Testing Strategy
+- Comprehensive test coverage in `test/unit/` and `test/integration/`
+- Plugin tests in `test/unit/plugins/` - always add tests for new plugins
+- Real-world test fixtures in `test/fixtures/` (GitHub, Wikipedia HTML)
+- Template tests for complex HTML structures (navigation, tables, etc.)
+- Always run tests after making changes to ensure backward compatibility
 
-Please reference the following docs:
+## Plugin Development
 
-- @docs/plugin-api.md
-- @docs/plugins.md
-- @docs/plugin-api.md
+When creating new plugins:
+1. Use CSS selectors from `src/libs/query-selector.ts` for element matching
+2. Implement memory-efficient patterns (immediate callbacks vs. collecting data)
+3. Add comprehensive tests covering edge cases and real-world scenarios
+4. Follow existing plugin patterns in `src/plugins/` directory
+5. Export from `src/plugins.ts` for public API access

README.md

@@ -99,16 +99,16 @@ pnpm add mdream
 ### Usage
 
 Mdream provides two utils for working with HTML, both will process content as a stream.
-- `syncHtmlToMarkdown`: Useful if you already have the entire HTML payload you want to convert.
+- `htmlToMarkdown`: Useful if you already have the entire HTML payload you want to convert.
 - `streamHtmlToMarkdown`: Best practice if you are fetching or reading from a local file.
 
 **Convert existing HTML**
 
 ```ts
-import { syncHtmlToMarkdown } from 'mdream'
+import { htmlToMarkdown } from 'mdream'
 
 // Simple conversion
-const markdown = syncHtmlToMarkdown('<h1>Hello World</h1>')
+const markdown = htmlToMarkdown('<h1>Hello World</h1>')
 console.log(markdown) // # Hello World
 ````
 
@@ -138,7 +138,7 @@ for await (const chunk of markdownGenerator) {
 Mdream now features a powerful plugin system that allows you to customize and extend the HTML-to-Markdown conversion process.
 
 ```ts
-import { createPlugin, filterUnsupportedTags, syncHtmlToMarkdown, withTailwind } from 'mdream'
+import { createPlugin, filterUnsupportedTags, htmlToMarkdown, withTailwind } from 'mdream'
 
 // Create a custom plugin
 const myPlugin = createPlugin({
@@ -153,7 +153,7 @@ const myPlugin = createPlugin({
 
 // Use multiple plugins together
 const html = '<div role="alert" class="font-bold">Important message</div>'
-const markdown = syncHtmlToMarkdown(html, {
+const markdown = htmlToMarkdown(html, {
   plugins: [
     withTailwind(), // Apply Tailwind class processing
     filterUnsupportedTags(), // Filter out unsupported tags
@@ -169,7 +169,7 @@ console.log(markdown) // "⚠️ **Important message** ⚠️"
 Extract specific elements and their content during HTML processing for data analysis or content discovery:
 
 ```ts
-import { extractionPlugin, syncHtmlToMarkdown } from 'mdream'
+import { extractionPlugin, htmlToMarkdown } from 'mdream'
 
 const html = `
   <article>
@@ -190,7 +190,7 @@ const plugin = extractionPlugin({
   }
 })
 
-syncHtmlToMarkdown(html, { plugins: [plugin] })
+htmlToMarkdown(html, { plugins: [plugin] })
 ```
 
 The extraction plugin provides memory-efficient element extraction with full text content and attributes, perfect for SEO analysis, content discovery, and data mining.

bench/bundle/src/await-fetch.ts

@@ -1,6 +1,6 @@
 import { writeFile } from 'node:fs/promises'
 import { resolve } from 'node:path'
-import { syncHtmlToMarkdown } from '../../../src'
+import { htmlToMarkdown } from '../../../src'
 
 async function run() {
   // read times to run it from command line argument
@@ -12,7 +12,7 @@ async function run() {
     // create a read stream for ../elon.html
     // create a read stream for ../elon.html
     const html = await response.text()
-    await writeFile(resolve(import.meta.dirname, '../dist/wiki.md'), await syncHtmlToMarkdown(html), { encoding: 'utf-8' })
+    await writeFile(resolve(import.meta.dirname, '../dist/wiki.md'), await htmlToMarkdown(html), { encoding: 'utf-8' })
   }
   const end = performance.now()
   const duration = end - start

bench/bundle/src/await.ts

@@ -1,6 +1,6 @@
 import { readFile, writeFile } from 'node:fs/promises'
 import { resolve } from 'node:path'
-import { syncHtmlToMarkdown } from '../../../src'
+import { htmlToMarkdown } from '../../../src'
 
 async function run() {
   // read times to run it from command line argument
@@ -11,7 +11,7 @@ async function run() {
     const html = await readFile(resolve(import.meta.dirname, '../wiki.html'), { encoding: 'utf-8' })
     logMemoryUsage('before await creation')
     const start = performance.now()
-    const converted = syncHtmlToMarkdown(html)
+    const converted = htmlToMarkdown(html)
     const end = performance.now()
     const duration = end - start
     // eslint-disable-next-line no-console

bench/bundle/src/minimal.ts

@@ -1,4 +1,4 @@
-import { syncHtmlToMarkdown } from '../../../src'
+import { htmlToMarkdown } from '../../../src'
 
 function run() {
   // Full usage with all core features
@@ -17,7 +17,7 @@ function run() {
 <img src="image.jpg" alt="Image description">
 <p>Another paragraph.</p>
   `
-  const markdown = syncHtmlToMarkdown(html)
+  const markdown = htmlToMarkdown(html)
 
   process.stdout.write(markdown)
 }

bench/bundle/src/string.ts

@@ -1,4 +1,4 @@
-import { syncHtmlToMarkdown } from '../../../src'
+import { htmlToMarkdown } from '../../../src'
 
 const html = `<!DOCTYPE html>
 <!-- saved from url=(0039)https://en.wikipedia.org/wiki/Elon_Musk -->
@@ -3527,7 +3527,7 @@ function run() {
   // extend the timings
   for (let i = 0; i < times; i++) {
     // eslint-disable-next-line no-console
-    console.log((syncHtmlToMarkdown(html)).length)
+    console.log((htmlToMarkdown(html)).length)
   }
   const end = performance.now()
   const duration = end - start

scripts/crawl.ts

@@ -1,7 +1,7 @@
 import { existsSync, mkdirSync } from 'node:fs'
 import { writeFile } from 'node:fs/promises'
 import { PlaywrightCrawler } from 'crawlee'
-import { syncHtmlToMarkdown } from '../src/index'
+import { htmlToMarkdown } from '../src/index'
 import { withMinimalPreset } from '../src/preset/minimal'
 
 // CheerioCrawler crawls the web using HTTP requests
@@ -13,7 +13,7 @@ const crawler = new PlaywrightCrawler({
     const html = await page.innerHTML('html')
     log.info('HTML length', { url: request.loadedUrl, length: html.length })
     const now = new Date()
-    const md = syncHtmlToMarkdown(html, withMinimalPreset({
+    const md = htmlToMarkdown(html, withMinimalPreset({
       origin: new URL(request.loadedUrl).origin,
     }))
     log.info('Processed html -> md in', { url: request.loadedUrl, time: new Date() - now })

src/index.ts

@@ -1,7 +1,7 @@
 import type { HTMLToMarkdownOptions, MdreamRuntimeState } from './types'
 import { processPartialHTMLToMarkdown } from './parser'
 
-export function syncHtmlToMarkdown(
+export function htmlToMarkdown(
   html: string,
   options: HTMLToMarkdownOptions = {},
 ): string {

test/unit/malformed-html.test.ts

@@ -1,27 +1,27 @@
 import { describe, it } from 'vitest'
-import { syncHtmlToMarkdown } from '../../src/index.js'
+import { htmlToMarkdown } from '../../src/index.js'
 
 describe.skip('malformed html', () => {
   it('correctly tracks element depth in nested structures', () => {
     it('handles incorrectly nested tags that overlap', () => {
       const html = '<p><strong>Bold text <em>Bold and italic</strong> just italic</em></p>'
-      const markdown = syncHtmlToMarkdown(html)
+      const markdown = htmlToMarkdown(html)
 
       // The parser should maintain emphasis even though tags are improperly nested
       expect(markdown).toContain('**Bold text *Bold and italic** just italic*')
     })
 
     it('recovers from malformed attributes in tags', () => {
       const html = '<a href="https://example.com" title="missing quote>Link text</a>'
-      const markdown = syncHtmlToMarkdown(html)
+      const markdown = htmlToMarkdown(html)
 
       // The parser should still create a link despite the malformed attribute
       expect(markdown).toContain('[Link text](https://example.com)')
     })
 
     it('handles broken HTML comments appropriately', () => {
       const html = '<!-- This comment is not closed <p>This paragraph should be visible</p>'
-      const markdown = syncHtmlToMarkdown(html)
+      const markdown = htmlToMarkdown(html)
 
       // The parser should still process content after a broken comment
       expect(markdown).toContain('This paragraph should be visible')

test/unit/nodes/blockquote.test.ts

@@ -1,35 +1,35 @@
 import { describe, expect, it } from 'vitest'
-import { syncHtmlToMarkdown } from '../../../src/index.js'
+import { htmlToMarkdown } from '../../../src/index.js'
 
 describe('blockquotes', () => {
   it('converts blockquotes', () => {
     const html = '<blockquote>This is a quote</blockquote>'
-    const markdown = syncHtmlToMarkdown(html)
+    const markdown = htmlToMarkdown(html)
     expect(markdown).toBe('> This is a quote')
   })
 
   it('handles nested blockquotes', () => {
     const html = '<blockquote>Outer quote<blockquote>Inner quote</blockquote></blockquote>'
-    const markdown = syncHtmlToMarkdown(html)
+    const markdown = htmlToMarkdown(html)
     expect(markdown).toBe('> Outer quote\n> > Inner quote')
   })
 
   it.skip('handles blockquotes with paragraphs', () => {
     const html = '<blockquote><p>First paragraph</p><p>Second paragraph</p></blockquote>'
-    const markdown = syncHtmlToMarkdown(html)
+    const markdown = htmlToMarkdown(html)
     expect(markdown).toBe('> First paragraph\n> Second paragraph')
   })
 
   it('handles complex nested blockquotes', () => {
     const html = '<blockquote><p>Outer paragraph</p><blockquote><p>Inner paragraph</p></blockquote></blockquote>'
-    const markdown = syncHtmlToMarkdown(html)
+    const markdown = htmlToMarkdown(html)
 
     expect(markdown).toBe('> Outer paragraph\n> > Inner paragraph')
   })
   // test for > A quote with an ![image](image.jpg) inside.
   it('handles blockquotes with images', () => {
     const html = '<blockquote>This is a quote with an <img src="image.jpg" alt="image"></blockquote>'
-    const markdown = syncHtmlToMarkdown(html)
+    const markdown = htmlToMarkdown(html)
     expect(markdown).toBe('> This is a quote with an ![image](image.jpg)')
   })
 })