encoding/json: performance slower than expected #5683

Open
gopherbot opened this Issue Jun 11, 2013 · 16 comments

Projects

None yet

8 participants

@gopherbot

by reid.write:

STR:
1. clone the git repository here: git@github.com:tarasglek/jsonbench.git
2. Generate some sample JSON data using the instructions in the README
3. Run the go json benchmark which is in gojson/src/jsonbench/json.go

What is the expected output?
I expected to see performance roughly in line with Java (using Jackson for json
parsing). On my test machine, the Java benchmark results in the following:
Processing 436525928 bytes took 5491 ms (75.82 MB/s)

What do you see instead?
Significantly slower performance, using the same input file:
Duration: 27.497043818s, 15.14 MB/s

Which compiler are you using (5g, 6g, 8g, gccgo)?
Not sure

Which operating system are you using?
Linux dell-ubuntu 3.2.0-45-generic #70-Ubuntu SMP Wed May 29 20:12:06 UTC 2013 x86_64
x86_64 x86_64 GNU/Linux

Which version are you using?  (run 'go version')
go version go1.1 linux/amd64
@rsc
Contributor
rsc commented Jun 14, 2013

Comment 1:

If you want us to investigate further, please attach a data file and a program.
I tried looking in your github repo but I couldn't figure out how to generate a data
file (enable telemetry means nothing to me) and it's probably good for us to have the
same data file anyway. 
I did look at the sources. Your Java program is just tokenizing the JSON, not building a
data structure. Go is actually building a data structure for you. So you are comparing
apples and oranges.
A more equal comparison would be to have Go unmarshal into
var x struct{}
dec.Decode(&x)
which will parse the JSON but throw away all the data. 
In practice, you should be parsing into a real struct anyway. I think you'll find that
case is much nicer in Go than in Java, and probably competitive in speed.

Status changed to WaitingForReply.

@gopherbot

Comment 2 by reid.write:

Thanks for the feedback!
You're right, it's much faster with the unmarshaling approach you recommended:
Duration: 10.034308451s, 47.98 MB/s
I will update the Java test to be more apples to apples.  The Java version is basically
only doing JSON Validation, so I'll modify it to parse the actual document structure.
In the meantime, I've added a new make target `make sample-data` that will generate a
~500MB test file, as well as a few more make targets to simplify running the various
tests.
Again, Go's performance when actually parsing the objects is:
Duration: 35.149537439s, 14.27 MB/s
For reference, here is the performance for a few of the other languages/runtimes on my
machine:
Javascript (spidermonkey):
39.550669327341836MB/s 525822000bytes in 12.679s
Python (simplejson):
37 MB/s 525831000bytes in 13seconds
C++ (rapidjson):
163 MB/s 525831000 bytes in 3 seconds
@davecheney
Contributor

Comment 3:

@reid - is there anything left to do on this ticket ?
@gopherbot

Comment 4 by reid.write:

I've made changes to the Java benchmark to make for a more apples-to-apples comparison. 
Go is still significantly slower than most of the other versions, and it should be
fairly straightforward to actually run the tests now (make sample-data; make java; make
python; make go)
If someone is able to look into improving json parsing performance, that would be much
appreciated!
@adg
Contributor
adg commented Jul 4, 2013

Comment 5:

If you don't publish your benchmark data and methodology (code), then your complaints
aren't actionable.
Anyone working on this needs to be making the same comparison. Saying "make it faster"
is not helpful.
@alberts
Contributor
alberts commented Jul 5, 2013

Comment 6:

It's right there in the first line, isn't it?
https://github.com/tarasglek/jsonbench
@gopherbot

Comment 7 by reid.write:

The motivation for filing this issue is that we are logging terabytes of Firefox
performance data in (compressed) JSON format and we'd really like to be able to process
& analyze this data set using a modern language like Go instead of Java / C++.
Here are the latest steps to reproduce:
1. clone (or update) the git repository here: git clone
git@github.com:tarasglek/jsonbench.git
2. Generate some sample JSON data: make sample-data
3. Run the go json benchmark: make go
Optionally, you can run some of the other language benchmarks using "make python", "make
java", etc.
Here are the results I get on my test machine:
make go: 14.36 MB/s in 34.932531708s
make python_simplejson: 39 MB/s 525831000bytes in 12seconds
make java: 48.82 MB/s 525822000 bytes in 10.27 s
Note: The # bytes is different for java because it's not counting end-of-line characters.
@rsc
Contributor
rsc commented Jul 9, 2013

Comment 8:

You are measuring something that is likely not relevant to the final
program, namely unmarshaling into a generic data type
(map[string]interface{}).
Almost any Go program doing JSON for data structure processing unmarshals
into a struct. Even if the struct lists every field in the input, that form
will be more compact in memory and therefore faster to create.
Russ
@gopherbot

Comment 9 by runner.mei:

I have the same feeling, I found the problem in the json/stream.go, as following:
   155  func (enc *Encoder) Encode(v interface{}) error {
   156      if enc.err != nil {
   157          return enc.err
   158      }
   159      enc.e.Reset()
   160      err := enc.e.marshal(v)
   161      if err != nil {
   162          return err
   163      }
   164  
   165      // Terminate each value with a newline.
   166      // This makes the output look a little nicer
   167      // when debugging, and some kind of space
   168      // is required if the encoded value was a number,
   169      // so that the reader knows there aren't more
   170      // digits coming.
   171      enc.e.WriteByte('\n')
   172  
---------------------------- here ----------------------------------
   173      if _, err = enc.w.Write(enc.e.Bytes()); err != nil {
   174          enc.err = err
   175      }
   176      return err
   177  }
In general the "w" is the buffer,  Write method had an unnecessary memory copy
@rsc
Contributor
rsc commented Nov 27, 2013

Comment 10:

Labels changed: added go1.3maybe.

@rsc
Contributor
rsc commented Dec 4, 2013

Comment 11:

Labels changed: added release-none, removed go1.3maybe.

@rsc
Contributor
rsc commented Dec 4, 2013

Comment 12:

Labels changed: added repo-main.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@mschuett mschuett referenced this issue in DECK36/go-log2gelf Sep 23, 2015
Open

CPU usage really high #3

@gaurish
Contributor
gaurish commented Dec 17, 2015

is there any work planned on this?

@bradfitz
Member

Nobody is working on it.

@kevinburke
Contributor

FWIW, I've resurrected the package in question, added structs so it's not just parsing into a struct{}, and added benchmarks around it that follow the format of the benchmarks in the standard library. Those are here: https://github.com/kevinburke/jsonbench/blob/master/go/bench_test.go

@kevinburke kevinburke added a commit to kevinburke/go that referenced this issue Jun 27, 2016
@kevinburke kevinburke encoding/json: Use a lookup table for safe characters
The previous check for characters inside of a JSON string that needed
to be escaped performed seven different boolean comparisons before
determining that a ASCII character did not need to be escaped. Most
characters do not need to be escaped, so this check can be done in a
more performant way.

Use the same strategy as the unicode package for precomputing a range
of characters that need to be escaped, then do a single lookup into a
character array to determine whether the character needs escaping.

On an AWS c4.large node:

$ benchstat benchmarks/master-bench benchmarks/json-table-bench
name                   old time/op    new time/op     delta
CodeEncoder-2            19.0ms ± 0%     15.5ms ± 1%  -18.16%        (p=0.000 n=19+20)
CodeMarshal-2            20.1ms ± 1%     16.8ms ± 2%  -16.35%        (p=0.000 n=20+21)
CodeDecoder-2            49.3ms ± 1%     49.5ms ± 2%     ~           (p=0.498 n=16+20)
DecoderStream-2           416ns ± 0%      416ns ± 1%     ~           (p=0.978 n=19+19)
CodeUnmarshal-2          51.0ms ± 1%     50.9ms ± 1%     ~           (p=0.490 n=19+17)
CodeUnmarshalReuse-2     48.5ms ± 2%     48.5ms ± 2%     ~           (p=0.989 n=20+19)
UnmarshalString-2         541ns ± 1%      532ns ± 1%   -1.75%        (p=0.000 n=20+21)
UnmarshalFloat64-2        485ns ± 1%      481ns ± 1%   -0.92%        (p=0.000 n=20+21)
UnmarshalInt64-2          429ns ± 1%      427ns ± 1%   -0.49%        (p=0.000 n=19+20)
Issue10335-2              631ns ± 1%      619ns ± 1%   -1.84%        (p=0.000 n=20+20)
NumberIsValid-2          19.1ns ± 0%     19.1ns ± 0%     ~     (all samples are equal)
NumberIsValidRegexp-2     689ns ± 1%      690ns ± 0%     ~           (p=0.150 n=20+20)
SkipValue-2              14.0ms ± 0%     14.0ms ± 0%   -0.05%        (p=0.000 n=18+18)
EncoderEncode-2           525ns ± 2%      512ns ± 1%   -2.33%        (p=0.000 n=20+18)

name                   old speed      new speed       delta
CodeEncoder-2           102MB/s ± 0%    125MB/s ± 1%  +22.20%        (p=0.000 n=19+20)
CodeMarshal-2          96.6MB/s ± 1%  115.6MB/s ± 2%  +19.56%        (p=0.000 n=20+21)
CodeDecoder-2          39.3MB/s ± 1%   39.2MB/s ± 2%     ~           (p=0.464 n=16+20)
CodeUnmarshal-2        38.1MB/s ± 1%   38.1MB/s ± 1%     ~           (p=0.525 n=19+17)
SkipValue-2             143MB/s ± 0%    143MB/s ± 0%   +0.05%        (p=0.000 n=18+18)

I also took the data set reported in #5683 (browser
telemetry data from Mozilla), added named structs for
the data set, and turned it into a proper benchmark:
https://github.com/kevinburke/jsonbench/blob/master/go/bench_test.go

The results from that test are similarly encouraging. On a 64-bit
Mac:

$ benchstat benchmarks/master-benchmark benchmarks/json-table-benchmark
name              old time/op    new time/op    delta
CodeMarshal-4       1.19ms ± 2%    1.08ms ± 2%   -9.33%  (p=0.000 n=21+17)
Unmarshal-4         3.09ms ± 3%    3.06ms ± 1%   -0.83%  (p=0.027 n=22+17)
UnmarshalReuse-4    3.04ms ± 1%    3.04ms ± 1%     ~     (p=0.169 n=20+15)

name              old speed      new speed      delta
CodeMarshal-4     80.3MB/s ± 1%  88.5MB/s ± 1%  +10.29%  (p=0.000 n=21+17)
Unmarshal-4       31.0MB/s ± 2%  31.2MB/s ± 1%   +0.83%  (p=0.025 n=22+17)

On the c4.large:

$ benchstat benchmarks/master-bench benchmarks/json-table-bench
name              old time/op    new time/op    delta
CodeMarshal-2       1.10ms ± 1%    0.98ms ± 1%  -10.12%  (p=0.000 n=20+54)
Unmarshal-2         2.82ms ± 1%    2.79ms ± 0%   -1.09%  (p=0.000 n=20+51)
UnmarshalReuse-2    2.80ms ± 0%    2.77ms ± 0%   -1.03%  (p=0.000 n=20+52)

name              old speed      new speed      delta
CodeMarshal-2     87.3MB/s ± 1%  97.1MB/s ± 1%  +11.27%  (p=0.000 n=20+54)
Unmarshal-2       33.9MB/s ± 1%  34.2MB/s ± 0%   +1.10%  (p=0.000 n=20+51)

For what it's worth, I tried other heuristics - short circuiting the
conditional for common ASCII characters, for example:

if (b >= 63 && b != 92) || (b >= 39 && b <= 59) || (rest of the conditional)

This offered a speedup around 7-9%, not as large as the submitted
change.

Change-Id: Idcf88f7b93bfcd1164cdd6a585160b7e407a0d9b
9f3fab4
@gopherbot

CL https://golang.org/cl/24466 mentions this issue.

@gopherbot gopherbot pushed a commit that referenced this issue Sep 8, 2016
@kevinburke @dsnet kevinburke + dsnet encoding/json: Use a lookup table for safe characters
The previous check for characters inside of a JSON string that needed
to be escaped performed seven different boolean comparisons before
determining that a ASCII character did not need to be escaped. Most
characters do not need to be escaped, so this check can be done in a
more performant way.

Use the same strategy as the unicode package for precomputing a range
of characters that need to be escaped, then do a single lookup into a
character array to determine whether the character needs escaping.

On an AWS c4.large node:

$ benchstat benchmarks/master-bench benchmarks/json-table-bench
name                   old time/op    new time/op     delta
CodeEncoder-2            19.0ms ± 0%     15.5ms ± 1%  -18.16%        (p=0.000 n=19+20)
CodeMarshal-2            20.1ms ± 1%     16.8ms ± 2%  -16.35%        (p=0.000 n=20+21)
CodeDecoder-2            49.3ms ± 1%     49.5ms ± 2%     ~           (p=0.498 n=16+20)
DecoderStream-2           416ns ± 0%      416ns ± 1%     ~           (p=0.978 n=19+19)
CodeUnmarshal-2          51.0ms ± 1%     50.9ms ± 1%     ~           (p=0.490 n=19+17)
CodeUnmarshalReuse-2     48.5ms ± 2%     48.5ms ± 2%     ~           (p=0.989 n=20+19)
UnmarshalString-2         541ns ± 1%      532ns ± 1%   -1.75%        (p=0.000 n=20+21)
UnmarshalFloat64-2        485ns ± 1%      481ns ± 1%   -0.92%        (p=0.000 n=20+21)
UnmarshalInt64-2          429ns ± 1%      427ns ± 1%   -0.49%        (p=0.000 n=19+20)
Issue10335-2              631ns ± 1%      619ns ± 1%   -1.84%        (p=0.000 n=20+20)
NumberIsValid-2          19.1ns ± 0%     19.1ns ± 0%     ~     (all samples are equal)
NumberIsValidRegexp-2     689ns ± 1%      690ns ± 0%     ~           (p=0.150 n=20+20)
SkipValue-2              14.0ms ± 0%     14.0ms ± 0%   -0.05%        (p=0.000 n=18+18)
EncoderEncode-2           525ns ± 2%      512ns ± 1%   -2.33%        (p=0.000 n=20+18)

name                   old speed      new speed       delta
CodeEncoder-2           102MB/s ± 0%    125MB/s ± 1%  +22.20%        (p=0.000 n=19+20)
CodeMarshal-2          96.6MB/s ± 1%  115.6MB/s ± 2%  +19.56%        (p=0.000 n=20+21)
CodeDecoder-2          39.3MB/s ± 1%   39.2MB/s ± 2%     ~           (p=0.464 n=16+20)
CodeUnmarshal-2        38.1MB/s ± 1%   38.1MB/s ± 1%     ~           (p=0.525 n=19+17)
SkipValue-2             143MB/s ± 0%    143MB/s ± 0%   +0.05%        (p=0.000 n=18+18)

I also took the data set reported in #5683 (browser
telemetry data from Mozilla), added named structs for
the data set, and turned it into a proper benchmark:
https://github.com/kevinburke/jsonbench/blob/master/go/bench_test.go

The results from that test are similarly encouraging. On a 64-bit
Mac:

$ benchstat benchmarks/master-benchmark benchmarks/json-table-benchmark
name              old time/op    new time/op    delta
CodeMarshal-4       1.19ms ± 2%    1.08ms ± 2%   -9.33%  (p=0.000 n=21+17)
Unmarshal-4         3.09ms ± 3%    3.06ms ± 1%   -0.83%  (p=0.027 n=22+17)
UnmarshalReuse-4    3.04ms ± 1%    3.04ms ± 1%     ~     (p=0.169 n=20+15)

name              old speed      new speed      delta
CodeMarshal-4     80.3MB/s ± 1%  88.5MB/s ± 1%  +10.29%  (p=0.000 n=21+17)
Unmarshal-4       31.0MB/s ± 2%  31.2MB/s ± 1%   +0.83%  (p=0.025 n=22+17)

On the c4.large:

$ benchstat benchmarks/master-bench benchmarks/json-table-bench
name              old time/op    new time/op    delta
CodeMarshal-2       1.10ms ± 1%    0.98ms ± 1%  -10.12%  (p=0.000 n=20+54)
Unmarshal-2         2.82ms ± 1%    2.79ms ± 0%   -1.09%  (p=0.000 n=20+51)
UnmarshalReuse-2    2.80ms ± 0%    2.77ms ± 0%   -1.03%  (p=0.000 n=20+52)

name              old speed      new speed      delta
CodeMarshal-2     87.3MB/s ± 1%  97.1MB/s ± 1%  +11.27%  (p=0.000 n=20+54)
Unmarshal-2       33.9MB/s ± 1%  34.2MB/s ± 0%   +1.10%  (p=0.000 n=20+51)

For what it's worth, I tried other heuristics - short circuiting the
conditional for common ASCII characters, for example:

if (b >= 63 && b != 92) || (b >= 39 && b <= 59) || (rest of the conditional)

This offered a speedup around 7-9%, not as large as the submitted
change.

Change-Id: Idcf88f7b93bfcd1164cdd6a585160b7e407a0d9b
Reviewed-on: https://go-review.googlesource.com/24466
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
ed8f207
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment