The latest updates on your projects. Learn more about Vercel for Git ↗︎

I haven't run any benchmarks on this, but I'm curious how much of a slowdown this adds, assuming no extensions. Do you have any measurements?

As an alternative that might be more flexible, but more restructuring... I wonder what you think about extracting each of the parser/renderer steps into an array of "extension-like objects".

I.e., instead of:

      beforeCode();
      // code
      if (token = this.tokenizer.code(src)) {
       ...
       beforeFences();

       // fences
      if (token = this.tokenizer.fences(src)) {
      ... 

something like (pseudocode):

const tokenizers = {
  code : ()=>{ if (token = this.tokenizer.code(src)) ... },
  fences : ()=>{ if (token = this.tokenizer.fences(src)) ... },
  ...
}

for ([tokenizerName, tokenizerFunction] in tokenizers) {
  runExtensionBefore(tokenizerName);
  tokenizerFunction();
}

I'm fairly certain there is some speed impact calling the tokenizers from an array/map/object like this but it might make the whole project more flexible for this type of "extension position" customization.

I did try to move the tokenizers to some sort of array but there were two problems.

1. Some tokenizers require a lot of extra logic that is not easy to move out of the lexer.
2. I couldn't think of an easy way to provide the array to the user so they can pick where to put their tokenizers

I think just adding a position property like this is the easiest for the user, and even though it is a lot of boilerplate we are not likely going to be adding or removing tokenizers from the lexer.

I did try to move the tokenizers to some sort of array but there were two problems.

Ok, fair enough. Looking back, this is what I tried to do back in 2021 and I think we concluded arrays just add too much slowdown anyway. #1872

Also looking back, we had discussed a "before" parameter 4 years ago. Hoping it wasn't also a lag issue that made us drop it #2043 (comment)

Remaining comments:

Ya, these 3 are in the TODOs section in this PRs description. I will work on them soon.

Also looking back, we had discussed a "before" parameter 4 years ago. Hoping it wasn't also a lag issue that made us drop it

Looks like we just figured we would get it done when it was actually needed