How to tokenize multiline constructs? How to subtokenize their nested content?

[vadistic]

Hello, I'm hoping for a bit of a tip how to tokenize multiline constructs.

I was trying to "guess this" for long hours, but it escapes me, could I ask for few tips? :)

Let's say I want to capture multiline content between custom <% and %> tags. Content inside should be subtokenized with all other available constructs and have containers (eg text wrapped in paragraphs).

<%

Some **inner content**

%>

// but it should also handle

<%
Some **inner content**
%>

// and

<% Some **inner content** %>

I'm writing simple tokenizer like this: https://gist.github.com/vadistic/d9d3bd683bd6227e36ffc1a4b7203574 and using it with micromark via mdast-util-from/to-markdown.

---

Q1: What kind of hooks should I use for such case?

I'm guessing flow?

The text works, but gets null/eof after single line. And tbh I cannot experiment with behavior of other kinds, because I cannot come up with working example :/

---

Q2: Why all non-text hooks produce those errors for me? What I'm doing wrong?

// with document

https://github.com/micromark/micromark/blob/main/lib/initialize/document.js#L184

TypeError: Cannot read property 'call' of undefined
      at exitContainers (../../node_modules/micromark/dist/initialize/document.js:132:28)
      at flowStart (../../node_modules/micromark/dist/initialize/document.js:75:7)
      at go (../../node_modules/micromark/dist/util/create-tokenizer.js:136:13)
      at main (../../node_modules/micromark/dist/util/create-tokenizer.js:130:9)
      at Object.write (../../node_modules/micromark/dist/util/create-tokenizer.js:70:5)
      at Object.fromMarkdown [as default] (../../node_modules/mdast-util-from-markdown/dist/index.js:26:34)

// with flow

https://github.com/micromark/micromark/blob/main/lib/util/subtokenize.js#L167

TypeError: Cannot read property '1' of undefined
      at subcontent (../../node_modules/micromark/dist/util/subtokenize.js:125:35)
      at subtokenize (../../node_modules/micromark/dist/util/subtokenize.js:61:23)
      at postprocess (../../node_modules/micromark/dist/postprocess.js:7:11)
      at Object.fromMarkdown [as default] (../../node_modules/mdast-util-from-markdown/dist/index.js:25:5)

---

Q3: How to subtokenize content between tags?

Q3.1: How to subtokenize content between tags using all available constructs? (with flow constructs)

The goal would be to render any available syntax inside those custom tags.

Q3.2: And how I would prevent this if I wanted the opposite? To skip any nested tokenization?

The goal would be to prevent parsing any nested content if it contained invalid (non-markdown) code.

Q3.3: How to subtokenize content as "text", as is my customTag already served as paragraph container?

The goal would be eg. something like rendering mdx string properties <Component prop="bold **value"**"/> but those cannot be paragraphs.

I know I can hack this on mdast level, but I want to try to have exact ast from the start. This is just an example of broader scope :)

**Q3.4: what's the default behavior if tokenizer jsut consumes code?

---

Q4: What construct options should I be using?

My best guess is concrete: true ?

Could provide me like with 3 words how lazy / interuptible / concrete / partial affect execution?

---

Thank you!

[wooorm]

“Containers” come from the margin, such as a block quote or a list item. So, “indent based”.
If you have a start and end line, it’s flow. Code (fenced) and HTML (flow) are much more similar to what you’re doing.
Or, in fact, the directive “container”, which is actually flow: https://github.com/micromark/micromark-extension-directive/blob/29eb2a025d177ee02682c0a7e0cc142a4aaa6633/lib/syntax.js#L12.

The text works, but gets null/eof after single line

Set concrete: true on your construct: https://github.com/micromark/micromark-extension-directive/blob/29eb2a025d177ee02682c0a7e0cc142a4aaa6633/lib/tokenize-directive-container.js#L4.

Q2: Why all non-text hooks produce those errors for me? What I'm doing wrong?

That error indicates that the lines aren’t linked properly. Tip: use the debugging features: https://github.com/micromark/micromark#size--debug.

Q3.1: How to subtokenize content between tags using all available constructs? (with flow constructs)

Can you elaborate more on why you want content in <% %>?

Q3.2: And how I would prevent this if I wanted the opposite? To skip any nested tokenization?

That seems more likely. See fenced code for an example.

Q3.3: How to subtokenize content as "text", as is my customTag already served as paragraph container?

Use text chunks tokens with a contentType, similar to how ATX heading do that:

micromark/lib/tokenize/heading-atx.js

Lines 63 to 69 in c8c644a

	text = {
	type: types.chunkText,
	start: events[contentStart][1].start,
	end: events[contentEnd][1].end,
	// @ts-expect-error Constants are fine to assign.
	contentType: constants.contentTypeText
	}

Q4: What construct options should I be using.

Yep.

Lazy means that lines don‘t have to be indented nicely in containers. Only content has that:

- asdasd
this is still part of the content, and doesn’t close the blockquote

Certain things can or cannot interrupt another construct, here is an example of indented code not interrupting a paragraph:

asd
    asd

Partial means that it’s not a “whole” construct, just a small nameless subroutine used to check if something matches.
For example, a link title is partial, or a list item indent. But a whole link and a whole list item are not.

[vadistic]

Thanks, you're really great!

also: I’m working right now on exposing, typing, and documenting all the extension places. It might be better to wait a bit, as it would safe you some time. On the other hand, I’d welcome all feedback

That really, really great! I know this project is few levels below public apis, but for core of something so widely used as remark it will really benefit with a bit of documentation. I already reverse-engineered big part of it it the hard way (with a bit of help from your side), but let's think about other users :)

Btw. I mostly done with my use-case, I'm not doing anything too smart, just needed to mostly ignore some weird, incompatible syntax so it parses and serializes back.

Can you elaborate more on why you want content in <% %>?

So it can work as custom container with whatever meaning. It needs to capture group of content (one unist node with everything inside as children[]) but the inner stuff should still parse/serialize/render as-it-was.

I know it's basically the feature of custom directive container, but I need it with custom syntax (I will probably copy-paste and modify that source if I do not figure it out^^).

---

I'm going to wait a bit and see if I get this working before accepting this answer, but it's already great. Thanks for debugger tip!

[wooorm]

So it can work as custom container with whatever meaning. It needs to capture group of content (one unist node with everything inside as children[]) but the inner stuff should still parse/serialize/render as-it-was.

What is in the children? It sounds like it can contain arbitrary text?

This task may be either impossible, if it is a template language that runs before markdown.
Or, it’s very close to https://github.com/micromark/micromark-extension-mdx-expression

[vadistic]

It can be both. Two different cases.

• If it's arbitrary language I need to disable tokenized (like code block)
• And if it's valid markdown I need to make sure it's also tokenized (like mdx expression)

Thus was this question how do I actually enable/disable nested tokenization inside "content" of some construct, what's the "switch" and what's the default behavior :)

[wooorm]

anything is valid markdown: https://github.com/micromark/micromark#grammar.
How do you expect to know whether something is or isn’t markdown?

[vadistic]

Different kind tags/syntax. I think I wanted to simplify this question and was not clear enough on this, sorry.

a) [syntax1] not-markdown [/syntax1]
b) [syntax2] markdown [/syntax2]

This is not the issue here, let me not take more of your time :)