[remark-gfm] Parsing long tables is very slow

[Maniload]

Hey 👋 I'm relatively new to using remark, so I'm not sure if I'm doing something wrong, but I noticed that parsing long tables with remark-gfm can get very slow. Tables with around 400 rows begin to take over one second to parse. In my (probably rare) use case I have a table with ~2000 rows, which takes around two minutes to parse.

I wrote a simple script to reproduce this:

import { remark } from "remark";
import remarkGfm from "remark-gfm";

let input = "Column 1 | Column 2 | Column 3\n--- | --- | ---";

for (let rows = 1; rows <= 1000; rows++) {
    input += "\nCell 1 | Cell 2 | Cell 3";

    console.time(`Rows: ${rows}`);

    remark().use(remarkGfm).processSync(input);

    console.timeEnd(`Rows: ${rows}`);
}

Is there something I can do to speed up the parsing or is this an issue with remark-gfm?

[ChristianMurphy]

Welcome @ManuelDevelopment!

A few things to note, your example doesn't just parse a thousand line table.
It parses increasingly large tables a thousand times, re-instantiating remark each time, and likely some garbage collection from the JS engine.

It's also worth noting that long and complex content takes longer to parse than simple and short content, period. 🙂

In terms of what can be done to trim extra time.
If you don't need any remark plugins, and just want GFM to be converted to HTML, you could use a lower level library offered by the same team behind remark, micromark
https://github.com/micromark/micromark#github-flavored-markdown-gfm

micromark example with timing: https://stackblitz.com/edit/github-fhpurw?file=src%2Findex.js
remark example with timing: https://stackblitz.com/edit/github-fhpurw-yulrbu?file=src%2Findex.js

That said, in my testing, in the sandbox (which has extra overhead over native node.js) both remark and micromark take under 30 seconds to process a table with 2000 rows.

[wooorm]

Hey! GFM tables are tough to parse, they are different than other markdown constructs and require looking ahead further.
That being said, there are likely things to improve where they are parsed: https://github.com/micromark/micromark-extension-gfm-table. Perhaps you can profile and help there?

[Maniload]

Thanks for the pointer! I tried profiling this and it appears that most time is spent with garbage collection. I'm not really experienced in profiling JS, so I'm not really sure where to go from here. Do you have an idea what could be causing this?

[wooorm]

Do you have an idea what could be causing this?

I do not have a precise idea.

You can for example try different versions of micromark-extension-gfm-table and micromark to understand if this is something that was introduced at some point or if it was already present from the start.
You can also try different markdown input, to see if the problem might be lines or columns or long cells or what.
You can change states to switch to the nok state, to see if that magically solves everything (of course, nok stops parsing, so it also breaks everything).

You can read in particular the Architecture section in the micromark readme for more on how the project works.

[shtse8]

it took 300ms to parse 187 rows table. causing it hard to use it in production with remark-gfm@3.0.1 and micromark@3.0.10

[wooorm]

There is an open issue. Please comment there, if it adds anything, or help work on a solution!

[charliematters]

I'm coming from a duplicate (ish) bug where I'm trying to just parse text and no tables. I'm getting these results:

micromark-extension-gfm-table@1.0.3

Processing without gfmTable: 30.86ms
Processing with gfmTable: 34.455ms

micromark-extension-gfm-table@1.0.4

Processing without gfmTable: 31.696ms
Processing with gfmTable: 10.622s

So I'm assuming the regression happened in 1.0.4?

[charliematters]

I notice that going back to 1.0.3 still has the behaviour I reported here: micromark/micromark-extension-gfm-table#4. Maybe it was the fix for my original bug which broke it? That would be some karma!

[pjkaufman]

I am not sure if this would help, but is it possible to look for the table separator row first and then if something matches that, check for a header row, validate that it has a proper row count and then find remaining tables rows? It requires being able to check the previous and next lines of text, but it could speed things up if it is trying to match any possible table values without first finding something that has to be present to have a table.

Maybe this has already been done or has a flaw, but I thought I would suggest it just in case it would be helpful.

[wooorm]

No, the architecture is per line. You can’t look for a line, and then look for the previous line.
Though, being able to do that, I don’t think it would make things faster: two lines need to be checked anyway, and then it has to be checked that they have the same number of cells.