Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize JSON->MsgPack stream conversion #725

Merged
merged 8 commits into from
Sep 15, 2019
Merged

Conversation

xerial
Copy link
Member

@xerial xerial commented Sep 15, 2019

  • Uses two-pass algorithm (fill empty MAP32/ARRAY32 headers in the first pass, then fill them with the real headers later after the size of arrays and maps can be determined). This will save one memory copy.
  • Added json_stream benchmark to see the performance.
    • JSON -> JSONValue -> MsgaPack conversion is not so bad for small JSON
    • JSON -> two-pass algorithm -> MsgPack is a clear winner for large JSON. For a small single JSON object, no significant difference exists.
Benchmark                    Mode  Cnt  Score    Error  Units
IntArraySON.jsonValue        avgt   10  0.010 ±  0.001  ms/op
IntArraySON.nested           avgt   10  0.011 ±  0.001  ms/op
IntArraySON.twoPass          avgt   10  0.018 ±  0.001  ms/op
TwitterJSON.jsonValue        avgt   10  5.338 ±  0.914  ms/op
TwitterJSON.nested           avgt   10  8.017 ±  1.082  ms/op
TwitterJSON.twoPass          avgt   10  4.743 ±  1.117  ms/op
TwitterSingleJSON.jsonValue  avgt   10  0.029 ±  0.007  ms/op
TwitterSingleJSON.nested     avgt   10  0.046 ±  0.018  ms/op
TwitterSingleJSON.twoPass    avgt   10  0.026 ±  0.006  ms/op

@codecov
Copy link

codecov bot commented Sep 15, 2019

Codecov Report

Merging #725 into master will decrease coverage by 0.52%.
The diff coverage is 85.91%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #725      +/-   ##
==========================================
- Coverage   83.29%   82.76%   -0.53%     
==========================================
  Files         217      218       +1     
  Lines        8523     8588      +65     
  Branches      598      603       +5     
==========================================
+ Hits         7099     7108       +9     
- Misses       1424     1480      +56
Impacted Files Coverage Δ
...main/scala/wvlet/airframe/msgpack/spi/Packer.scala 50% <ø> (ø) ⬆️
...rframe/msgpack/json/NestedMessagePackBuilder.scala 0% <0%> (ø)
...main/scala/wvlet/airframe/codec/MessageCodec.scala 84.31% <100%> (ø) ⬆️
...cala/wvlet/airframe/msgpack/spi/OffsetPacker.scala 98.13% <100%> (+0.02%) ⬆️
...scala/wvlet/airframe/msgpack/spi/WriteCursor.scala 93.33% <100%> (+0.47%) ⬆️
...scala/wvlet/airframe/msgpack/spi/MessagePack.scala 80% <100%> (ø) ⬆️
...scala/wvlet/airframe/msgpack/impl/PackerImpl.scala 64.44% <100%> (+0.8%) ⬆️
...rc/main/scala/wvlet/airframe/codec/JSONCodec.scala 89.58% <66.66%> (ø) ⬆️
...rframe/msgpack/json/StreamMessagePackBuilder.scala 89.65% <89.65%> (ø)
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d992e0f...3541f7d. Read the comment docs.

@xerial xerial merged commit 6a07fb4 into wvlet:master Sep 15, 2019
@xerial xerial deleted the json-stream-opt branch September 15, 2019 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant