Fix Fleet server performance bottlenecks due to JSON unmarshal

[getvictor]

Problem: The Go json/v1 API we are using is at least 2X slower at unmarshaling than pretty much any other library. We do a lot of JSON unmarshaling between osquery results, vulnerability processing, etc.

• json/v2 realistically won't be out for at least a year. We could use it as is, but its API is unstable.
• I recommend we try https://github.com/goccy/go-json to speed up JSON unmarshal where needed. It is popular (used by 177K open source repos) and API-compatible with json/v1. Once json/v2 is out, we can start using that.

Example performance issue: #24390

Another example from an old vuln processing pprof run:

[mostlikelee]

associated risk (thx @mna):

It's interesting to investigate, but the use of unsafe is concerning (IIRC, the json/v2 aims for speedups without using unsafe) and there are 8 reports of panic issues in the first page of issues alone (6 "panic", 1 "exception", 1"segmentation" in the title), and 2 high mem-cpu usage : https://github.com/goccy/go-json/issues

Some have accompaniying PR open for a year and a half. It's... not a good look. Would be good to also verify if they take shortcuts like in fastjson, e.g. to skip bytes assuming a well-formed JSON and not report invalid JSON (we accept user-provided JSON, it's important that our unmarshaler checks its validity).
Some folks have lots of imagination when it comes to topping benchmarks.

[getvictor]

Another extremely popular JSON library is https://github.com/bytedance/sonic and it is also not memory-safe.

If we must have memory safety, then I recommend using https://github.com/go-json-experiment/json

There is another memory-safe alternative, but it doesn't seem very popular:

• https://github.com/sugawarayuuta/sonnet

[jacobshandling]

In case it might be a decent option, I ran across the simd-json library and this write up about simdjson

[getvictor]

In case it might be a decent option, I ran across the simd-json library and this write up about simdjson

@jacobshandling Thank you for sharing. I took a look at it. It only works on Intel/AMD x86 chips, so it might not be a good fit for our M1 dev machines.

[lukeheath]

Needs to be updated to new process: https://fleetdm.com/handbook/engineering#create-an-engineering-initiated-story.

[getvictor]

Closing this. We will automatically get better performance once json/v2 is released in an upcoming Go release.

[fleet-release]

Faster parsing flow,
Like a swift cloud city breeze,
Fleet's data streams glow.