Improved Concrete Syntax Trees

[benrbray]

Subject of the feature

I'm in the process of migrating my markdown editor to use remark/micromark instead of markdown-it. One of my goals is not to change the formatting style of my users' input files, at least if I can help it.

At the moment, the micromark tokenizers seem not to record information that might help reconstruct the original input Markdown in cases where Markdown has redundancies:

• _ vs * for emphasis
• ATX headings vs setext headings
• * vs - vs + for unordered lists
• * vs - vs = for hrule / thematic breaks, as well as the length of the string used to indicate the break

I'm not terribly interested in preserving superfluous whitespace the user might have, but it would be nice to at least preserve their preferences for emphasis / heading / list syntax. For instance, I personally like to use * for regular lists and +/- for pro/con lists, and at the moment there's no way to preserve that information.

• Are there any plans for improved concrete syntax tree support in the future?
• If not would PRs be welcome to record syntactic information on syntax tree nodes, as well as in the corresponding serializers?
• Otherwise, any advice for implementing this feature as a set of plugins would be appreciated!

Thanks!

Problem

Expected behavior

Alternatives

[wooorm]

^-- this is my fault I think

[wooorm]

Are there any plans for improved concrete syntax tree support in the future?

It would mean a different syntax tree that is hard to manage as it is so verbose.
micromark was made to allow making a CST. There have been several discussions about it over the years: https://github.com/search?q=CST+user%3Awooorm+user%3Amicromark+user%3Aremarkjs+user%3Asyntax-tree&type=Issues
It would be cool to have, but it’s also a ginormous effort to design a concrete syntax tree, and would affect many things in unified.

If not would PRs be welcome to record syntactic information on syntax tree nodes, as well as in the corresponding serializers?

I’m against adding such information on abstract syntax tree nodes, as I think ASTs by definition should not have such information: they’re supposed to be lossy and easy to handle.

Otherwise, any advice for implementing this feature as a set of plugins would be appreciated!

You can detect such information, similar to how remark-lint detects such information, in a plugin. It could record that information and turn it into an object that mdast-util-to-markdown expects

[github-actions]

Hi! Thanks for reaching out! Because we treat issues as our backlog, we close issues that are questions since they don’t represent a task to be completed.

See our support docs for how and where to ask questions.

Thanks,
— bb