Use serde-transcode to optimize JSON formatting #362
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A read operation might get an "interrupted" error, in which case the correct behavior is (usually) to try again as if nothing happened. When we read a compressed stream we check whether the underlying reader received an error. But we should only check whether it received an error for the latest read, so that we can ignore these interrupts properly. This is the only place I noticed where interrupts were handled improperly. AFAIK this can't happen in reality because we don't install signal handlers, but it's good practice.
Instead of parsing the JSON input into a complicated heap value before writing it out we can write it out as we parse it. On a ridiculously large input this gives me a 5× speedup.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Followup to #361 (which I meant to look at but forgot).
Instead of parsing the JSON input into a
serde::Valuewe can useserde-transcodeto serialize straight from the input to the output. On an articial very large JSON file I use for testing this ends up five times faster. (Maybe slightly faster thanjsonxfwas?) I got this trick from jsonxf's serde benchmark.(The rest of the PR looked great. Thanks, @zuisong!)
I also included a tiny theoretical bugfix for decompression that I wrote a while ago but didn't think was worth a PR, see the explanation in 398d567