Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob: tooBig is too small #27635

Closed
bf opened this issue Sep 12, 2018 · 50 comments
Closed

encoding/gob: tooBig is too small #27635

bf opened this issue Sep 12, 2018 · 50 comments
Assignees
Milestone

Comments

@bf
Copy link

@bf bf commented Sep 12, 2018

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

arch linux with 16gb ram

What did you do?

I parse large CSV files for social media analytics, creating in-memory data structures from the data. When persisting these data structures to disk using gob, I receive gob: encoder: message too big error from this code:

encoder := gob.NewEncoder(file)
err := encoder.Encode(MapData)

One set of input data is a 4GB csv file which results in a data structure that is 200MB when saved with gob. The second set of input data is a 15GB csv file which should result in a 1200MB file when saved with gob, but I receive the error above.

The error is thrown at a check if the data structure is larger than the tooBig constant at

if messageLen >= tooBig {

The tooBig constant is originally defined with a comment TODO: Make this adjustable here

// TODO: make this adjustable?
const tooBig = 1 << 30

What did you expect to see?

I expect gob to save large data structures of ~2GB that dont get close to my RAM limits of 16GB without any problems.

What did you see instead?

I got the gob: encoder: message too big error.

I fixed the error by compiling my own go and changing the tooBig check in encoder.go to 1<<31. Afterwards the data structure from the large 15GB CSV file was successfully saved by gob in a 1500MB gob file.

Therefore I suggest this change could be done for all users, as the existing TODO comment in the golang source code suggests

// TODO: make this adjustable?

@bf bf changed the title Increase tooBig from 1<<30 to 1<<31 Increase tooBig from 1<<30 to 1<<31 for gob encoder Sep 12, 2018
@agnivade
Copy link
Contributor

@agnivade agnivade commented Sep 12, 2018

/cc @robpike

@mvdan
Copy link
Member

@mvdan mvdan commented Sep 12, 2018

This was first introduced in 9c3fc83, because earlier it was easy to get encoding/gob to panic with large amounts of data due to overflows.

My initial reaction would be "we can just bump this limit on 64-bit", or "we can just use 64-bit sized integers in the encoder and decoder", although I don't know the internals of the gob package that well.

The original issue was #8084. Interestingly enough, it happened on amd64.

I guess it's fine to double the limit while we don't have a better long-term solution, but I'll leave that to Rob to decide.

@mvdan mvdan changed the title Increase tooBig from 1<<30 to 1<<31 for gob encoder encoding/gob: tooBig is too small Sep 12, 2018
@mvdan mvdan added the NeedsDecision label Sep 12, 2018
@bf
Copy link
Author

@bf bf commented Sep 12, 2018

@mvdan thanks for your input. I tried to looking through the gob source but I can't really make out what kind of integers are used.

But what I can tell you is that I changed the data structure from map[int]map[int]map[int]map[uint64]bool to map[int8]map[int8]map[int8]map[uint64]bool in order to save disk space and potentially work around the error message. I found out the size of the gob dump file did not change when using int8 instead of the standard int (which afaik is at least int32).

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Sep 12, 2018
@magical
Copy link
Contributor

@magical magical commented Sep 13, 2018

@bf FYI, gob uses a variable-length format to encode integers, so it makes sense that changing the integer type in your data structure did not affect the output size.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 19, 2018

Maybe tooBig should be bigger on 64-bit. On 32-bit I think 1GB is fine. Thoughts, @robpike?

@robpike robpike self-assigned this Sep 20, 2018
@robpike
Copy link
Contributor

@robpike robpike commented Sep 20, 2018

Sure.

@rsc rsc added the NeedsFix label Sep 26, 2018
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

16 similar comments
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

22 similar comments
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2018

Change https://golang.org/cl/143678 mentions this issue: encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines

@gopherbot gopherbot closed this in 3bf9b77 Oct 22, 2018
@josharian
Copy link
Contributor

@josharian josharian commented Oct 22, 2018

@dmitshur what happened with gopherbot here?

@agnivade
Copy link
Contributor

@agnivade agnivade commented Oct 22, 2018

Github outage. All comments got duplicated.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 22, 2018

Hopefully this was the only notification email party that Gopherbot had in the issue tracker :)

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Oct 22, 2018

Indeed. I kept an eye on gopherbot logs (among other builder-related services that touch GitHub) during the GitHub semi-outage, and didn't spot it doing anything visibly bad. (See #28320 for what I did find, but it's unrelated to this.)

Gopherbot determines if it has acted by looking if its comment already exists on an issue, so if GitHub reports the comment as not being there (as was the case), it can't help but think it hasn't acted yet and repeat the action. In other words, it relies on GitHub working.

GitHub does offer a status API. If we really had to take action, we could add code that pauses gopherbot when https://status.github.com/api reports that GitHub is in poor state. But as far as I can see, gopherbot didn't do anything too outrageous and so we don't have to do that.

@golang golang locked and limited conversation to collaborators Oct 22, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.