Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob, encoding/json: add streaming interface #14140

Closed
rgooch opened this issue Jan 28, 2016 · 5 comments
Closed

encoding/gob, encoding/json: add streaming interface #14140

rgooch opened this issue Jan 28, 2016 · 5 comments

Comments

@rgooch
Copy link

@rgooch rgooch commented Jan 28, 2016

When encoding a large data structure (such as a tree representing a file-system), the Encode() methods first marshal the encoded data into a []byte buffer and then write the buffer to the writer. Similarly, the Decode() methods require a buffer to unmarshal from. These buffers must be allocated (and grown) and typically more than double the memory consumption of the process. This is limiting for applications which are sized to the machine/container.

I propose that the interfaces are extended to support streaming encoding/decoding of data. For each type (except slices and maps), the data would be separately marshaled and written. For slices and maps, each entry would be marshaled and written, rather the slice/map as a whole.

Some API suggestions/alternatives:
func Encode(w io.Writer, v interface{}) error
func NewStreamingEncoder(w io.Writer) *Encoder
func (enc *Encoder) StreamingEncode(v interface{}) error

@robpike
Copy link
Contributor

@robpike robpike commented Jan 28, 2016

While I understand the desire, one of the strengths of gobs is that the receiver always knows how much buffer to allocate because the message has a header that says how long it is. That can't be done without storing the whole message in memory in the encoder.

@rgooch
Copy link
Author

@rgooch rgooch commented Jan 28, 2016

Yes, I see how that's been helpful for the net/rpc package: knowing the size means that you can multiplex RPCs over a single connection. But there are plenty of situations in which the receiver doesn't care how big the message will be. It's even more the case with the current version of the language since there is no reliable way to know if an allocation will fail, so a receiver can't make an informed decision on whether to accept or drop that 1 GiB incoming transmission.

I'm not suggesting changing the current properties. I'm suggesting an extension that makes the package useful for large data structures. It could just as well be new packages like "sgob" and "sjson" but that would result in a lot of duplicate code.

@ianlancetaylor ianlancetaylor changed the title Add streaming interface to encoding/gob and encoding/json encoding/gob, encoding/json: add streaming interface Jan 28, 2016
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jan 28, 2016
@tejorupan
Copy link

@tejorupan tejorupan commented Jan 30, 2016

#6569 net/rpc: Streaming API for RPC

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 26, 2017

Related to #11046 for encoding/json.

@mvdan
Copy link
Member

@mvdan mvdan commented Nov 24, 2020

#33714 goes into a lot more detail (and even an implemented design) for json. For the sake of keeping discussion in one place, I'm going to close this issue in favor of that one.

I realise this issue is also about encoding/gob, though it does not include a specific proposal and it hasn't gotten any traction. I think someone should open an issue specific to encoding/gob if they still strongly feel that it should have streaming support.

@mvdan mvdan closed this Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants