Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob: slow map decoding #19525

Open
opennota opened this issue Mar 13, 2017 · 4 comments

Comments

@opennota
Copy link

commented Mar 13, 2017

What version of Go are you using (go version)?

go version go1.8 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/opennota/gocode"
GORACE=""
GOROOT="/home/opennota/go"
GOTOOLDIR="/home/opennota/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build966707479=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I tried to encode a big map to gob

  1. as a map.
  2. as slices of keys and values

and decode the encoded data back. In the first case the data decodes almost 3x slower.

https://play.golang.org/p/ZpHC-4I9p1

Output:

533.188157ms
180.698776ms

@bradfitz bradfitz added this to the Unplanned milestone Mar 13, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Mar 17, 2017

CL https://golang.org/cl/38317 mentions this issue.

gopherbot pushed a commit that referenced this issue Mar 24, 2017
encoding/gob: Speedup map decoding by reducing the allocations.
The improvementis achieved in encoding/gob/decode.go decodeMap by
allocate keyInstr and elemInstr only once and pass it to
decodeIntoValue, instead of allocating a new instance on every loop
cycle.

name                     old time/op  new time/op  delta
DecodeComplex128Slice-8  64.2µs ±10%  62.2µs ± 8%     ~     (p=0.686 n=4+4)
DecodeFloat64Slice-8     37.1µs ± 3%  36.5µs ± 5%     ~     (p=0.343 n=4+4)
DecodeInt32Slice-8       33.7µs ± 3%  32.7µs ± 4%     ~     (p=0.200 n=4+4)
DecodeStringSlice-8      59.7µs ± 5%  57.3µs ± 1%     ~     (p=0.114 n=4+4)
DecodeInterfaceSlice-8    543µs ± 7%   497µs ± 3%     ~     (p=0.057 n=4+4)
DecodeMap-8              3.78ms ± 8%  2.66ms ± 2%  -29.69%  (p=0.029 n=4+4)

Updates #19525

Change-Id: Iec5fa4530de76f0a70da5de8a129a567b4aa096e
Reviewed-on: https://go-review.googlesource.com/38317
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Apr 1, 2017

CL https://golang.org/cl/39203 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 9, 2017

CL https://golang.org/cl/40113 mentions this issue.

gopherbot pushed a commit that referenced this issue May 3, 2017
encoding/gob: use MakeMapWithSize when decoding map
This allows to pre-allocate the final size of the hashmap and avoid
re-allocating as we insert entries. Furthermore for the current
implementation of the hashmap it allows avoiding several rounds of
evacuating hashmap entries after each re-allocation.

DecodeComplex128Slice-8  51.9µs ± 1%  51.9µs ± 2%     ~     (p=0.797 n=30+29)
DecodeFloat64Slice-8     31.5µs ± 2%  31.6µs ± 2%     ~     (p=0.050 n=28+28)
DecodeInt32Slice-8       32.0µs ± 2%  31.9µs ± 3%     ~     (p=0.666 n=29+28)
DecodeStringSlice-8      57.7µs ± 2%  57.8µs ± 3%     ~     (p=0.780 n=27+30)
DecodeInterfaceSlice-8    498µs ± 2%   495µs ± 2%     ~     (p=0.070 n=28+29)
DecodeMap-8               300µs ± 2%   230µs ± 5%  -23.31%  (p=0.000 n=27+27)

Updates #19525

Change-Id: Ia7233da49f05bae7a86c064d9ecebca966f5f2f7
Reviewed-on: https://go-review.googlesource.com/40113
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit that referenced this issue May 26, 2017
encoding/gob: speedup decoding of maps by zeroing values
Instead of allocating a new reflect.Value object on every loop we zero it.

DecodeComplex128Slice-8  13.1µs ± 7%  13.2µs ± 8%     ~     (p=0.347 n=18+19)
DecodeFloat64Slice-8     8.13µs ± 5%  8.00µs ± 3%     ~     (p=0.168 n=20+19)
DecodeInt32Slice-8       8.27µs ± 5%  8.08µs ± 5%   -2.27%  (p=0.001 n=19+18)
DecodeStringSlice-8      17.9µs ±12%  17.8µs ±11%     ~     (p=0.989 n=20+19)
DecodeInterfaceSlice-8    163µs ±10%   159µs ± 4%     ~     (p=0.057 n=19+19)
DecodeMap-8               220µs ± 2%   183µs ± 1%  -17.07%  (p=0.000 n=19+18)

Updates #19525

Change-Id: I27f8edd4761787f6b9928d34cefa08a34a6e25b2
Reviewed-on: https://go-review.googlesource.com/39203
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@agnivade

This comment has been minimized.

Copy link
Member

commented Jan 4, 2018

Just an update on the nos. with latest tip as of today

328.972009ms
164.29887ms

From 3x to 2x. The gap is reducing :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.