How to handle “virtual spacing” in the CST?

[wooorm]

I’m going to post a couple of problems I foresee as I’m trying to wrap my head around what micromark will be.

Take the following example:

>␉␠indented.code("in a block quote")

It’s a block quote marker, followed by a tab (tabs are forced to be treated as four spaces).
The first “virtual space” of the tab is part of the block quote marker. The second three “virtual spaces” are part of the indent of the indented code.
One extra real space, and you’ve got a code indent of four spaces, making it a proper indented code, in a block quote.

How is that represented as tokens? In a CST?

[ChristianMurphy]

tabs are forced to be treated as four spaces

In a CST would it be encoded as four spaces?
Wouldn't it be preserved as a tab?

My understand of CSTs is that it will always reflect the literal text on the page.

[wooorm]

I think it would be present as a tab, so it’ll reflect the literal text of the page indeed, but there needs to be an “extra” thing that’s part of the the indented code as its indent.

Could maybe be an {type: 'indent', value: ' ', size: 4} (a node without only the one space, but with the size extra from the previous block quote tab?) 🤷‍♂️

[ChristianMurphy]

I think it would be present as a tab, so it’ll reflect the literal text of the page indeed, but there needs to be an “extra” thing that’s part of the the indented code as its indent

🤔 let's split this off into another discussion.

[ChristianMurphy]

It’s a block quote marker, followed by a tab (tabs are forced to be treated as four spaces).
The first “virtual space” of the tab is part of the block quote marker. The second three “virtual spaces” are part of the indent of the indented code.
One extra real space, and you’ve got a code indent of four spaces, making it a proper indented code, in a block quote.

I'd tend to expect the indentation always be in another node.
The spaces/tabs are needed to make it clear that > is in fact a block quote and not plain text.
But I'm not sure they are truly a part of the blockquote, but are their own indent node.
That indent node, could be a child of blockquote, but would still be it's own node.

[wooorm]

The spaces/tabs are needed to make it clear that > is in fact a block quote and not plain text.

Not quite, one space after > is optional. ># heading is a heading in a block quote.

That indent node, could be a child of blockquote, but would still be its own node.

Right! I’m thinking the nodes would look something like this: blockQuote[line[marker, indent, content]], where content is indentedCode[line[indent, content]].
The problem I’m foreseeing is what the values of indent in the block quote line and indent in the indented code line are.

[wooorm]

some more thoughts: The current attempt of micromark (the one checked in) is, I now believe, incorrect. It assumes Markdown can be parsed in blocks (which may work in an AST).
Markdown is instead line-based, I’m working on a new proposal, and I think virtual spacing should work something like this:

A line is made up of block continuation and block openings: >␉␠i is a blockquote marker token, followed by a whitespace token. Because the whitespace token is big enough, the rest of the line is part of indented code.

While tokenizing, this “big enough” can be accomplished by using a “conceptual” (non-real) character: a “Virtual space”.

Say we have: >␉␉i, the characters fed into the state machine are then: >, ␉, VS, VS, ␉, VS, VS, VS, i. Now, every whitespace character has a size of 1, so this whitespace token has a size of 7.
The blockquote “uses” 1 size of that, the indented code uses 4, leaving the last VS, VS characters. Those are part of the whitespace token, but can be used by something that compiles to HTML as a prefix before content: <blockquote><pre><code>␠␠i…

This does not affect tokens, if we’re doing something similar to mdast the whitespace token should look something like: {type: 'whitespace', value: '\t\t', characters: [9, -1, -1, 9, -1, -1, -1], size: 7, used: 5}

[wooorm]

This is now defined and solved in CMSM, by defining virtual spaces and content prefixes.