Strange bug(?) with this.lexer.blockTokens

[jmckalex]

I've encountered some strange behaviour in how this.lexer.blockTokens() processes content. Here's what happens: in my app, I have loaded a bunch of extensions, including marked-alert. Now, if I include the text:

> [!IMPORTANT]
> Crucial information necessary for 
> users to succeed.

as a block in the markdown file, it gets processed correctly and appears as an alert. However, if in a custom extension I've written I pass a string containing that input — "\n> [!IMPORTANT]\n> Crucial information necessary for\n> users to succeed.\n" — to this.lexer.blockTokens(), it gets processed as a blockquote. I can't figure out why this would happen. (I've tried swapping the order in which my extension and marked-alert are loaded, and it makes no difference.)

Can anyone please shed some light on this curious behaviour?

[UziTech]

Can you provide the code for your custom extension?

My guess is somewhere you are instantiating another version of the lexer that does not include the marked-alert extension

[jmckalex]

Hi Tony, Thanks for the quick response! Please find attached two files which define the custom extension. The extension I referred to is defined in 'markdown-demo.js', but that needs the code from 'extended-directives.js'. The extended-directives.js is based pretty closely on the marked-directives package, but I've hacked it so as to allow a custom tokeniser to be provided as well as a custom renderer. (This particular directive doesn’t take too much advantage of the custom tokeniser, but several other directives I've defined for my project do.) The extension is then installed in my main file as follows (notice that marked-alert is configured right before): import markedAlert from 'marked-alert'; marked.use(markedAlert()); marked_copy.use(markedAlert()); import createMarkdownDemo from './markdown-demo.js'; const markdownDemos = [ createMarkdownDemo(':::'), createMarkdownDemo('::::'), createMarkdownDemo(':::::'), createMarkdownDemo('::::::'), createMarkdownDemo(':::::::'), createMarkdownDemo(':::::::') ]; marked.use( createDirectives( markdownDemos ) ); This is all part of a much bigger project I intend to release once I finish cleaning up the code base. You can read a description of it here at the following link — and section 4.3 illustrates the failure of marked-alerts to work: https://jmckalex.org/software/jmarkdown/jmarkdown.html On 10 Apr 2025, at 14:51, Tony Brix ***@***.***> wrote: Can you provide the code for your custom extension? My guess is somewhere you are instantiating another version of the lexer that does not include the marked-alert extension — Reply to this email directly, view it on GitHub<#3658 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADFPKYFVJIYJQVM6EKPTZD32YZZMTAVCNFSM6AAAAAB23AUPH2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZZGIYTCMA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jmckalex]

Sorry - it looks like the files I attached got scrubbed. I've attached markdown-demo.js and extended-directives.js, bundled in a ZIP.

Here is the relevant (I think) portion of index.js:

import { createDirectives, presetDirectiveConfigs } from './extended-directives.js';
marked.use(createDirectives([
	...presetDirectiveConfigs,
	{ level: 'container', marker: '::::' },
	{ level: 'container', marker: ':::::' },
	{ level: 'container', marker: '::::::' },
	{ level: 'container', marker: ':::::::' },
	{ level: 'container', marker: '::::::::' }
]));

import markedAlert from 'marked-alert';
marked.use(markedAlert());

import createMarkdownDemo from './markdown-demo.js';
const markdownDemos = [
  createMarkdownDemo(':::'),
  createMarkdownDemo('::::'),
  createMarkdownDemo(':::::'),
  createMarkdownDemo('::::::'),
  createMarkdownDemo(':::::::'),
  createMarkdownDemo(':::::::')
];
marked.use( createDirectives( markdownDemos ) );

files.zip

[UziTech]

Can you create a minimal repo that reproduces this. It will be easier to find the issue if I can see a working example

[jmckalex]

Here's a pretty minimal repo. After installing the dependencies, if you run

node index-minimal.js minimal.md

The console will show the marked output. The relevant point is the beginning, where from the input

> [!NOTE]
> Highlights information that users 
> should take into account, even 
> when skimming.

:::markdown-demo
> [!NOTE]
> Highlights information that users 
> should take into account, even 
> when skimming.
:::

the alert is properly processed, but when the contents of the markdown-demo container are run through blockTokens it appears as a blockquote.

minimal-repo.zip

[UziTech]

The way marked-alert works is it uses walkTokens to change blockquotes to alert tokens when it finds an alert. Your marked demo directives add any child tokens to the property output which walkTokens doesn't know is a list of child tokens. if you change it to the property tokens or add childTokens: ["output"] (see docs) to the extension than walkTokens would know that the inner blockquote needs to change to an alert.

[jmckalex]

Many thanks — that fixed the problem!