New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob, encoding/json: add streaming interface #14140

Open
rgooch opened this Issue Jan 28, 2016 · 4 comments

Comments

Projects
None yet
4 participants
@rgooch

rgooch commented Jan 28, 2016

When encoding a large data structure (such as a tree representing a file-system), the Encode() methods first marshal the encoded data into a []byte buffer and then write the buffer to the writer. Similarly, the Decode() methods require a buffer to unmarshal from. These buffers must be allocated (and grown) and typically more than double the memory consumption of the process. This is limiting for applications which are sized to the machine/container.

I propose that the interfaces are extended to support streaming encoding/decoding of data. For each type (except slices and maps), the data would be separately marshaled and written. For slices and maps, each entry would be marshaled and written, rather the slice/map as a whole.

Some API suggestions/alternatives:
func Encode(w io.Writer, v interface{}) error
func NewStreamingEncoder(w io.Writer) *Encoder
func (enc *Encoder) StreamingEncode(v interface{}) error

@robpike

This comment has been minimized.

Contributor

robpike commented Jan 28, 2016

While I understand the desire, one of the strengths of gobs is that the receiver always knows how much buffer to allocate because the message has a header that says how long it is. That can't be done without storing the whole message in memory in the encoder.

@rgooch

This comment has been minimized.

rgooch commented Jan 28, 2016

Yes, I see how that's been helpful for the net/rpc package: knowing the size means that you can multiplex RPCs over a single connection. But there are plenty of situations in which the receiver doesn't care how big the message will be. It's even more the case with the current version of the language since there is no reliable way to know if an allocation will fail, so a receiver can't make an informed decision on whether to accept or drop that 1 GiB incoming transmission.

I'm not suggesting changing the current properties. I'm suggesting an extension that makes the package useful for large data structures. It could just as well be new packages like "sgob" and "sjson" but that would result in a lot of duplicate code.

@ianlancetaylor ianlancetaylor changed the title from Add streaming interface to encoding/gob and encoding/json to encoding/gob, encoding/json: add streaming interface Jan 28, 2016

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jan 28, 2016

@tejorupan

This comment has been minimized.

tejorupan commented Jan 30, 2016

#6569 net/rpc: Streaming API for RPC

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Oct 26, 2017

Related to #11046 for encoding/json.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment