Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob: Gob encoder's buffering all data in memory may cause oom #8316

Closed
gopherbot opened this issue Jul 2, 2014 · 5 comments
Closed

encoding/gob: Gob encoder's buffering all data in memory may cause oom #8316

gopherbot opened this issue Jul 2, 2014 · 5 comments

Comments

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 2, 2014

by daviddengcn:

My program is saving a big data structure that is about half the current memory. I found
`Encode.EncodeValue` will firstly buffer all the data to the memory (in enc.byteBuf),
then write it as a whole (by calling enc.writeMessage). This consumes almost the same
amout of memory as the value to encode. My program then occasionally panic because of
out of memory.

Can we make it better?

go version go1.3 linux/386
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 2, 2014

Comment 1:

Labels changed: added repo-main, release-none.

@robpike
Copy link
Contributor

@robpike robpike commented Jul 2, 2014

Comment 2:

Infeasible.

Status changed to Unfortunate.

@gopherbot
Copy link
Author

@gopherbot gopherbot commented Jul 2, 2014

Comment 3 by daviddengcn:

Can anybody technically explain why infeasible?
@robpike
Copy link
Contributor

@robpike robpike commented Jul 2, 2014

Comment 4:

Messages in the stream are preceded by their byte count for the benefit of the decoder.
Without major restructuring and overhead, there is no way to preserve compatibility
without keeping the encoded message in memory so its length can be transmitted first.
@gopherbot
Copy link
Author

@gopherbot gopherbot commented Jul 2, 2014

Comment 5 by daviddengcn:

OIC, thanks!
@golang golang locked and limited conversation to collaborators Jun 25, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.