Request: simd / AVX512 support

[Erotemic]

Based on recent benchmarks it looks like Json performance might greatly benefit from access to SIMD instructions.

https://www.phoronix.com/scan.php?page=article&item=simdjson-avx-512&num=2

I'm not sure how difficult support would be to add and maintain. Would the default pypi binaries need to have it disabled for compatibility? Would there need to be a custom pypi index for different cflags like their is with torch? Or would this be something that could be enabled/disabled at runtime?

[bwoodsend]

I'm guessing that it can go on PyPI. If I'm reading the internet correctly, this simd instruction is something that OpenMP uses under the hood and OpenMP is something that Cython makes a bit too easy so I've seen a few Cython libraries with prebuilt wheels on PyPI using OpenMP.

This does sound like a total nightmare though. ujson's code is already hard enough to work with that it's picking up memory bugs and CVEs - extremely low level multithreading is not going to help that.

[bwoodsend]

It's probably a lot easier to build a new Python wrapper for simdjson than adapt ujson to do whatever magic simdjson does.

[Erotemic]

That's a good idea.

[hugovk]

There are already three sets of Python bindings for simdjson listed at https://github.com/simdjson/simdjson#bindings-and-ports-of-simdjson