marked-rs

Rust port of marked v15.0.12, focused on reproducing marked.lexer() token output for Claude Code’s Markdown path:

marked.lexer(input) -> formatToken(...) -> ANSI string -> <Ansi>

This is intentionally not a generic CommonMark wrapper (pulldown-cmark / comrak / …). Behavior is guided by marked v15.0.12 and golden fixtures generated from that exact version.

Status

Layer	Status	Notes
Lexer (marked.lexer)	Done	147 golden fixtures match marked v15.0.12
Options (gfm / breaks / pedantic / disable_del)	Done	Claude Code’s del() override covered
Parser + HTML Renderer	Smoke	Optional; Claude Code does not use HTML output
Extension / hooks system	Out of scope	Only disable_del needed

Claude Code only consumes the token tree (never marked.parse() HTML). See docs/CLAUDE_CODE_INTEGRATION.md.

Layout (mirrors upstream)

src/
  lib.rs                 public API
  options.rs             MarkedOptions          ← MarkedOptions.ts / defaults.ts
  rules.rs               static regexes         ← rules.ts
  lexer.rs               block/inline dispatch  ← Lexer.ts
  tokenizer/
    block.rs             block tokenizers       ← Tokenizer.ts (block)
    inline.rs            inline tokenizers      ← Tokenizer.ts (inline)
  tokens.rs              Token enum             ← Tokens.ts
  helpers.rs             helpers                ← helpers.ts
  parser.rs              token walk (HTML)      ← Parser.ts
  renderer.rs            HTML emitters          ← Renderer.ts

Regex matching uses fancy-regex so JS-style lookahead / lookbehind / backreferences from marked can be ported directly.

API

use marked_rs::{lexer, lexer_with_options, marked, MarkedOptions, Token};

// Primary path (Claude Code): source → tokens
let tokens = lexer("# Hello");
assert!(matches!(tokens.tokens[0], Token::Heading { depth: 1, .. }));

// Claude Code equivalent of:
//   marked.use({ tokenizer: { del() { return undefined } } })
let tokens = lexer_with_options(
    "~~literal~~",
    MarkedOptions {
        disable_del: true,
        ..MarkedOptions::default()
    },
);

// Optional HTML path (smoke-level parity)
let html = marked("# Hello");
assert_eq!(html, "<h1>Hello</h1>\n");

TokensList serializes as a JSON array (same shape as JSON.stringify(marked.lexer(src))). Reference definitions stay on the Rust-side tokens.links map.

Golden tests

Fixtures: tests/fixtures/*.md
Goldens: tests/goldens/*.json (token JSON from marked v15.0.12)

Goldens are regenerated against a local checkout of the upstream marked repo (not a nested npm package in this crate):

# clone / checkout marked v15.0.12 somewhere, then:
export MARKED_SRC=/path/to/marked   # e.g. a git checkout of markedjs/marked @ v15.0.12
npx tsx tests/generate_goldens.mjs  # rewrites tests/goldens/*.json

cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test

tests/generate_goldens.mjs loads $MARKED_SRC/src/marked.ts (preferred, via tsx) or $MARKED_SRC/lib/marked.esm.js if already built.

Option suffixes in fixture names:

Suffix	Options
.disable_del.	disable_del = true
.breaks.	breaks = true
.gfm_off.	gfm = false
.pedantic.	pedantic = true

Upstream attribution

Behavior is derived from marked v15.0.12 (Tokens.ts, rules.ts, Lexer.ts, Tokenizer.ts, helpers.ts, …). marked is MIT licensed; this crate is a clean Rust reimplementation validated by golden tests.

Known gaps

See docs/KNOWN_DIFFERENCES.md. For Claude Code’s actual marked surface (lexer + disable_del + optional gfm: false), coverage is complete.