JSON encoder refactor

[chancancode]

Goals

• Replace ActiveSupport's pure-Ruby JSON encoder with one based on the json gem
• Maintain backwards compatibility (encode_json support) by moving the old encoder into a gem
• NOT introduce significant performance degradation
• Fix Rail's json gem incompatibilities (via JSON gem compatibility #12862)
• Fix as_json not being applied recursively

Nice to have

• Pluggable backend? (so that the encode_json gem can take advantage of that)
• Improved JSON performance (Benchmark: full stack, raw JSON)
• A technical explanation/spec for the all the relevant design (here)
• A user guide article explaining all these JSON related stuff

Misc TODOs

• Ask: can we fix the json version to >= 1.7.7? (Yes, Requires JSON gem version 1.7.7 or above as it contains an important security fix. #13069)
• Figure out what to do with this (bug), and people who overrides to_json in gerenral
• Check JSON ext compat: potentially JSON.generate throws error but MultiJson.dump does not sidekiq/sidekiq#1070
• Check JSON pure compat: bugs, track [json_pure] Caught Encoding::CompatibilityError incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) ruby/json#183
• Check JRuby compat
• Check RBX compat
• Check multi json compat...?
• Perf test: JSON API in discourse
• Perf test: compiling assets
• http://hashrocket.com/blog/posts/faster-json-generation-with-postgresql / https://github.com/JackC/json_api_bench
• Close ActiveRecord::Base.extend kills JSON performance #9212
• Check HashWithIndifferentAccess cannot be JSON.pretty_generate'd #12723

[chancancode]

Numers are in!

Full stack benchmark: this uses Discourse's bench.rb which uses apache bench to simulate real requests, so it goes through the whole stack. It's JSON-heavy because they dump the JSON data for the pages in the HTML output to "preload" the Ember data store without an additional HTTP request.

Raw JSON benchmark: this uses the same Discourse seed data generator and calculates the time it takes to dump various data.

Seems like the patched version is faster than both 4.0.1 and current master!

I also started a draft on a document that goes into details on the expected behaviour of the various components. (I swear it's not as scary as it looks!) The current implementation might not be 100% inline with the draft yet, but we can work on it in future PRs.

I do plan to turn that into a higher-level user guide at some point.

[chancancode]

The old encoder is extracted to here: https://github.com/vanruby/activesupport-json_encoder

[nixme]

@chancancode I just want to say thank you so much for this work 👏

[chancancode]

@nixme 💛

[defp]

👍

[chancancode]

4.1