Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

protoc-gen-go: fully generate marshaling and unmarshaling code #280

Open
awalterschulze opened this issue Jan 18, 2017 · 6 comments
Open

protoc-gen-go: fully generate marshaling and unmarshaling code #280

awalterschulze opened this issue Jan 18, 2017 · 6 comments

Comments

@awalterschulze
Copy link

@awalterschulze awalterschulze commented Jan 18, 2017

The Conversation started in #276

Is there a possibility of golang/protobuf starting to generate some marshaling and unmarshaling code?
I would recommend this over any use of unsafe, since then protocol buffers doesn't just play nice with appengine, but also gopherjs.
Also generated code is much faster, which is one of the reasons people flock to gogoprotobuf.
It would also be great if the generated code didn't depend on the proto package.
But this is just my personal wish, one less dependency for users of a package using protobufs.

  • Walter Schulze

We're looking into ways to speed up marshal and unmarshal. Entirely generated code seems unlikely to be the right end point (too much code, not enough performance win), but we're evaluating multiple points in the solution space. Code not depending on the proto package at all is an interesting idea, although that implies even more generated code. I don't think that was on our radar.

@awalterschulze
Copy link
Author

@awalterschulze awalterschulze commented Jan 18, 2017

I really think the amount of speed gained is significant enough to warrant the extra code generation for marshal and unmarshal.

Here is an independant benchmark
https://github.com/alecthomas/go_serialization_benchmarks

BenchmarkGoprotobufMarshal-8                2000000        746 ns/op      312 B/op        4 allocs/op
BenchmarkGoprotobufUnmarshal-8              2000000        978 ns/op      432 B/op        9 allocs/op
BenchmarkGogoprotobufMarshal-8             10000000        211 ns/op       64 B/op        1 allocs/op
BenchmarkGogoprotobufUnmarshal-8            5000000        289 ns/op       96 B/op        3 allocs/op

Also once you have those methods, the extra generated code required to not use the proto package is very little.

@rsc rsc changed the title Again: Why not generate marshaling and unmarshaling code generate marshaling and unmarshaling code Jan 18, 2017
@rsc
Copy link
Contributor

@rsc rsc commented Jan 18, 2017

@awalterschulze
Copy link
Author

@awalterschulze awalterschulze commented Jan 18, 2017

This surely cannot be news?

But if it is, this is gogoprotobuf, it does code generation (and other gadgets) for protobuffers for go, since 2013.
https://github.com/gogo/protobuf

It has users, including Kubernetes
https://github.com/gogo/protobuf#users

@awalterschulze
Copy link
Author

@awalterschulze awalterschulze commented Jan 18, 2017

One of the newer things you can do with gogoprotobuf.
Using this command you can still import github.com/golang/protobuf/proto, but get the generated marshaler, size and unmarshaler. So only your code generation steps depends on gogoprotobuf.

go install github.com/gogo/protobuf/protoc-gen-gofast
protoc --gofast_out=. my.proto

Maybe this can give you an idea of what generating code in golang/protobuf would be like.

@johanbrandhorst
Copy link
Member

@johanbrandhorst johanbrandhorst commented Feb 18, 2018

Bump, this discussion should be revisited, as it still has no satisfying conclusion. Recent discussion implies this still hasn't been closed as a possibility. What are the current thoughts on this?

@dsnet dsnet changed the title generate marshaling and unmarshaling code protoc-gen-go: fully generate marshaling and unmarshaling code Mar 8, 2018
@dsnet dsnet removed the enhancement label Jul 10, 2019
@dsnet dsnet added the proposal label Aug 7, 2019
@dsnet dsnet changed the title protoc-gen-go: fully generate marshaling and unmarshaling code proposal: protoc-gen-go: fully generate marshaling and unmarshaling code Aug 7, 2019
@dsnet dsnet changed the title proposal: protoc-gen-go: fully generate marshaling and unmarshaling code protoc-gen-go: fully generate marshaling and unmarshaling code Mar 4, 2020
@dsnet
Copy link
Member

@dsnet dsnet commented Oct 23, 2020

Wanted update this thread. @neild and I have talked on/off about this in the past. One of things we'd like to explore first before addressing this is to figure out whether to provide a streaming or scatter/gather API (see #609). One significant detriment with generated code is that the implementation easily grows stale unless users regenerate the code (which rarely occurs as has been observed).

Though it's not a hard ordering requirement, we would presumably want to know if we're going to go anywhere with streaming or scatter/gather APIs before doing generated serialization as the former would likely affect the internal APIs and implementation of the latter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants