Markdown::Lex

A Markdown lexer for text that is still being written. It turns a document into a flat list of typed blocks and styled spans, and hands them to whoever is drawing the screen: no HTML, no renderer, no dependencies, no opinions about colour.

It exists because a terminal chat client has to draw a model's answer while the tokens are still arriving, which means re-lexing the same growing string a few hundred times a second and never once getting an exception or a paragraph that flickers out of existence.

Synopsis

use Markdown::Lex;

for parse("# Title\n\nHello **world** and `code`.\n") -> $block {
    given $block {
        when Markdown::Lex::Heading {
            say "H{$block.level}: ", $block.inlines».text.join;
        }
        when Markdown::Lex::Para {
            for $block.inlines -> $span {
                print $span.bold ?? bold($span.text) !! $span.text;
            }
        }
        when Markdown::Lex::CodeFence {
            say "[{$block.lang // 'text'}]", $block.closed ?? '' !! ' (still writing)';
            .say for $block.lines;
        }
    }
}

The two promises

Total. parse never throws. Not on Str:U, not on the empty string, not on a document that is one unterminated code fence, not on ten thousand asterisks in a row. Whatever goes in, a List of blocks comes out.

Prefix-stable. Every prefix of a document is itself a valid document, so a consumer can do this and never think about it again:

my $accumulated = '';

react whenever $token-supply -> $chunk {
    $accumulated ~= $chunk;
    render(parse($accumulated));
}

Half-arrived constructs degrade to literal text rather than vanishing: **bo is a plain span until the closing asterisks turn up. The single exception is the code fence, which becomes a CodeFence with :!closed the moment it opens — because a renderer wants to draw the box while the code streams into it.

The blocks

Class	Attributes
Markdown::Lex::Heading	C<Int:D $.level> (1..6), C<@.inlines>
Markdown::Lex::Para	C<@.inlines>
Markdown::Lex::CodeFence	C<Str $.lang&gt;, C&lt;@.lines&gt;, C&lt;Bool:D $.closed>
Markdown::Lex::Bullet	C<Int:D $.depth&gt; (0-based), C&lt;Str:D $.marker>, C<@.inlines>
Markdown::Lex::Quote	C<@.inlines>
Markdown::Lex::Rule	(none)

All six are immutable, all six are a Markdown::Lex::Block, and all six have a .gist worth printing. use Markdown::Lex; exports one symbol — parse — and the classes are reachable fully qualified, which is what a given/when wants anyway.

The spans

@.inlines is a flat list of Markdown::Lex::Span, never a tree:

class Markdown::Lex::Span {
    has Str:D  $.text is required;
    has Bool:D $.bold   = False;
    has Bool:D $.italic = False;
    has Bool:D $.code   = False;
    has Str    $.link;              # undefined unless this span is a link
}

Nesting is expressed by combining flags, because that is exactly what a terminal can draw:

parse("**bold with *both* inside**").head.inlines;
# Span("bold with " :b) Span("both" :bi) Span(" inside" :b)

So there is no nesting depth limit and no recursion. Adjacent runs with identical flags are merged and an empty span is never emitted, which means a renderer can loop over @.inlines with no defensive checks at all.

What it lexes

ATX headings, fenced code (backticks or tildes, with an info word), bullets (-/*/+ and ordinals, with a depth taken from the indentation), block quotes, thematic breaks, and paragraphs that merge their soft line breaks into spaces. Inline: code spans, **bold**, *italic*, _italic_, __bold__, ***both*** and [text](destination).

Emphasis follows CommonMark's delimiter-run rules closely enough that the cases which bite in practice come out right — snake_case_name is not italic, 5 * 3 * 2 is arithmetic, *a **b** c* nests, and a leftover delimiter is literal text rather than a swallowed one.

What it does not lex

Setext headings, indented code blocks, tables, images, raw HTML, autolinks, footnotes, reference links, hard line breaks and backslash escapes. Nested block structure is out too: Bullet.depth is the only nesting a consumer gets.

Every one of them degrades to literal text, which is this module's only failure mode. Nothing is ever dropped.

The full list of divergences, with reasons, is in the module's own Pod: raku --doc lib/Markdown/Lex.rakumod, or just open the file — it opens with them.

Testing

prove6 -Ilib t/

No dependencies, so nothing to install first. The suite covers each block and inline construct, the CommonMark cases that are easy to get wrong, CRLF and lone carriage returns, Unicode from ZWJ emoji to combining accents to right-to-left text, and — the ones that matter most — a loop over every prefix of a torture document asserting that the lexer neither throws nor drops a single word, plus timing guards on five-thousand-character delimiter storms to catch a quadratic regression.

Author

Matt Doughty

License

Artistic-2.0