Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob: slow map decoding #19525

Open
opennota opened this issue Mar 13, 2017 · 4 comments
Open

encoding/gob: slow map decoding #19525

opennota opened this issue Mar 13, 2017 · 4 comments
Labels
Milestone

Comments

@opennota
Copy link

@opennota opennota commented Mar 13, 2017

What version of Go are you using (go version)?

go version go1.8 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/opennota/gocode"
GORACE=""
GOROOT="/home/opennota/go"
GOTOOLDIR="/home/opennota/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build966707479=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I tried to encode a big map to gob

  1. as a map.
  2. as slices of keys and values

and decode the encoded data back. In the first case the data decodes almost 3x slower.

https://play.golang.org/p/ZpHC-4I9p1

Output:

533.188157ms
180.698776ms
@bradfitz bradfitz added this to the Unplanned milestone Mar 13, 2017
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 17, 2017

CL https://golang.org/cl/38317 mentions this issue.

gopherbot pushed a commit that referenced this issue Mar 24, 2017
The improvementis achieved in encoding/gob/decode.go decodeMap by
allocate keyInstr and elemInstr only once and pass it to
decodeIntoValue, instead of allocating a new instance on every loop
cycle.

name                     old time/op  new time/op  delta
DecodeComplex128Slice-8  64.2µs ±10%  62.2µs ± 8%     ~     (p=0.686 n=4+4)
DecodeFloat64Slice-8     37.1µs ± 3%  36.5µs ± 5%     ~     (p=0.343 n=4+4)
DecodeInt32Slice-8       33.7µs ± 3%  32.7µs ± 4%     ~     (p=0.200 n=4+4)
DecodeStringSlice-8      59.7µs ± 5%  57.3µs ± 1%     ~     (p=0.114 n=4+4)
DecodeInterfaceSlice-8    543µs ± 7%   497µs ± 3%     ~     (p=0.057 n=4+4)
DecodeMap-8              3.78ms ± 8%  2.66ms ± 2%  -29.69%  (p=0.029 n=4+4)

Updates #19525

Change-Id: Iec5fa4530de76f0a70da5de8a129a567b4aa096e
Reviewed-on: https://go-review.googlesource.com/38317
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 1, 2017

CL https://golang.org/cl/39203 mentions this issue.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 9, 2017

CL https://golang.org/cl/40113 mentions this issue.

gopherbot pushed a commit that referenced this issue May 3, 2017
This allows to pre-allocate the final size of the hashmap and avoid
re-allocating as we insert entries. Furthermore for the current
implementation of the hashmap it allows avoiding several rounds of
evacuating hashmap entries after each re-allocation.

DecodeComplex128Slice-8  51.9µs ± 1%  51.9µs ± 2%     ~     (p=0.797 n=30+29)
DecodeFloat64Slice-8     31.5µs ± 2%  31.6µs ± 2%     ~     (p=0.050 n=28+28)
DecodeInt32Slice-8       32.0µs ± 2%  31.9µs ± 3%     ~     (p=0.666 n=29+28)
DecodeStringSlice-8      57.7µs ± 2%  57.8µs ± 3%     ~     (p=0.780 n=27+30)
DecodeInterfaceSlice-8    498µs ± 2%   495µs ± 2%     ~     (p=0.070 n=28+29)
DecodeMap-8               300µs ± 2%   230µs ± 5%  -23.31%  (p=0.000 n=27+27)

Updates #19525

Change-Id: Ia7233da49f05bae7a86c064d9ecebca966f5f2f7
Reviewed-on: https://go-review.googlesource.com/40113
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit that referenced this issue May 26, 2017
Instead of allocating a new reflect.Value object on every loop we zero it.

DecodeComplex128Slice-8  13.1µs ± 7%  13.2µs ± 8%     ~     (p=0.347 n=18+19)
DecodeFloat64Slice-8     8.13µs ± 5%  8.00µs ± 3%     ~     (p=0.168 n=20+19)
DecodeInt32Slice-8       8.27µs ± 5%  8.08µs ± 5%   -2.27%  (p=0.001 n=19+18)
DecodeStringSlice-8      17.9µs ±12%  17.8µs ±11%     ~     (p=0.989 n=20+19)
DecodeInterfaceSlice-8    163µs ±10%   159µs ± 4%     ~     (p=0.057 n=19+19)
DecodeMap-8               220µs ± 2%   183µs ± 1%  -17.07%  (p=0.000 n=19+18)

Updates #19525

Change-Id: I27f8edd4761787f6b9928d34cefa08a34a6e25b2
Reviewed-on: https://go-review.googlesource.com/39203
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@agnivade
Copy link
Contributor

@agnivade agnivade commented Jan 4, 2018

Just an update on the nos. with latest tip as of today

328.972009ms
164.29887ms

From 3x to 2x. The gap is reducing :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.