Better performance

[davesnx]

First I would love to profile q and try to understand where are the pain points, and I have no clue how to do that yet.

Assumption

The biggest operation is reading the file and parsing the JSON if the JSON is huge, this becomes more of a problem.

Exploration

I would expect OCaml multicore to help on the parallelization of parse/compile the user input while parse the JSON, that would speed up things, but I would love to parse the parts of the JSON that q needs, instead of parsing the entire file.

The other idea was improving the parsing of the JSON perse, there an exploration to that, which tries to make the JSON parsing a parallelizable operation, it's called "Stack Monoid" https://raphlinus.github.io/gpu/2020/09/05/stack-monoid.html.

Assumption

query-json's AST is recursive which makes the compilation to be recursive as well and it's tail-recursive, I heard that OCaml compiler optimizes the TL.

while jq, is a stack based described here and they are doing a lot of more work than query-json.

Exploration

See how the OCaml compiler optimizes our TL, and check that the recursion is TL.
Explain better how the jq Stack works.

Assumption

menhir it's a parser generator, which you write your rules in .mly files and OCaml creates the parser for you. It's obvious that having a parser written natively sounds that could be more optional in terms of performance.

Exploration

How much effort would be to turn the Parser.mly into a regular Reason file and how much it will improve performance.

[davesnx]

https://github.com/evanw/esbuild#benchmarks