simdjson

[v1gnesh]

Hi,

Hope you're doing well, whoever reads this!

I noticed a recent commit about JSON #1789
... and this made me wonder if you can/would consider simdjson and its Python bindings to speed things up with JSON.

[maartenbreddels]

Yeah, absolutely, as you may have seen, I already prepared the code to handle that: https://github.com/vaexio/vaex/pull/1789/files#diff-9e5185594b9c2651ff1a92481843ff26a9dace4c362d978e52b31937261e8767R6811

Although I think it should be a bit different. I think we should give it a dumps argument, which should default to None, and if None should use pandas, and otherwise it should we should set use_pandas=False. Also we should check it it really begins and ends with a '[' and ']'.

So we could have usage like:

import simdjson
df.export_json('large.json', dumps=simdjson.dumps)