Bug in latex example in tokenizer extension docs

[UlisseMini]

The latex example in the docs is broken, it doesn't handle foo $tex$ bar (foo disappears). This should be mentioned, or the docs should be fixed.

I spent a while trying to get latex like $this$ working without success,
I read #1538 but solution 1 was a hack putting math inside code blocks,
and solution 2 was complex and stateful.

IMO this is the kind of thing that should be easy. I like the example in the docs except it doesn't work ):

I read #1653 but the solution seems overly complicated, I can do something like that but IMO modifying inlineText shouldn't be necessary for this.
I don't understand the codebase though so maybe this is unavoidable because of how lexing is done.

I'd be happy to contribute a fix if I figure it out, or someone explains to me :)

[UziTech]

Ya the example in the docs is to show how to extend the tokenizer not necessarily how to implement latex. It also doesn't handle multiple $'s.

modifying inlineText shouldn't be necessary for this.

Ya we have been talking about ways to make that easier. Currently to make marked faster inlineText consumes everything up until it sees the potential start of a new token so when extending the tokenizer we need to extend inlineText to know what the new token start would be.

I'd be happy to contribute a fix if I figure it out

That would be great!

[UlisseMini]

I ended up using this workaround (for latex `$like this`)

marked.use({
  renderer: {
    codespan: (code) => {
      if (code[0] == '$') {
        return katex.renderToString(code.slice(1), {throwOnError: false})
      }

      return false
    }
  }
})

I mentioned the hole in the docs in #1949, haven't fixed it though ): reworking the lexer is a bit more than I'm willing to take on right now. though #1872 looks promising.

[UziTech]

fixed by #1949