Source maps

[jgm]

See commonmark/commonmark-spec#57, especially @zdne's comment.

Instead of starting and ending line and column for each element, we need to associate each element with a possibly non-contiguous range of positions in the source.

This is because CommonMark inline elements can be broken by indicators of block structure:

> *emphasized
> text*

Here the second > should not be considered part of the emphasized text, even though it occurs after the start of the emphasized text and before the end.

[jgm]

Note that this common format for source maps isn't useful for us here, because it correlates a textual original source with a textual generated source. We need to correlate positions in the original textual source with elements in the AST.

Something like:

sourcemap = [ [1, 3, 6], [2, 3, 5] ]

to indicate that the node occupies columns 3-6 of line 1 and columns 3-5 of line 2.

[puzrin]

I guess, you need this package https://github.com/mozilla/source-map . It's used everywhere to build soursemap data and includes support for incremental (nested) builds with flatten at the end. Don't know about other languages.

Though, that will significantly decrease speed and will require to rewrite regex replacements because those break positions. IMHO, it's better to rewrite parser first to be modular/pluggable.