Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proposal for introducing Protobuf serialization #22600

Merged
merged 1 commit into from
Mar 17, 2016
Merged

Proposal for introducing Protobuf serialization #22600

merged 1 commit into from
Mar 17, 2016

Conversation

smarterclayton
Copy link
Contributor

For the 1.3 timeframe.

Current status:

  • Generation tools are implemented and merged (can run hack/update-generated-protobuf.sh today)
  • Scripts are not enabled by default in dev process for updating/verifying protobuf
  • Serializer PR is open Add a protobuf serializer #20377
  • Watch framing is prototyped, but needs testing and framing has a few bugs
  • Client can be set to use protobuf via the ContentType header and server should support it

@smarterclayton
Copy link
Contributor Author

@kubernetes/sig-api-machinery @wojtek-t

@smarterclayton smarterclayton added the kind/design Categorizes issue or PR as related to design. label Mar 5, 2016
@k8s-github-robot
Copy link

Labelling this PR as size/L

@k8s-github-robot k8s-github-robot added kind/design Categorizes issue or PR as related to design. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 5, 2016
@smarterclayton smarterclayton force-pushed the protoproposal branch 2 times, most recently from 526590d to d416b56 Compare March 5, 2016 22:49
@k8s-github-robot
Copy link

Labelling this PR as size/XL

@k8s-github-robot k8s-github-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 5, 2016
@k8s-bot
Copy link

k8s-bot commented Mar 5, 2016

GCE e2e build/test passed for commit 6601d4a1f4ef56f9728db62e5960dcf4dbeea1c8.

@k8s-bot
Copy link

k8s-bot commented Mar 5, 2016

GCE e2e build/test passed for commit 526590db385ab8839f69d1e398d65a3165252e10.

@k8s-bot
Copy link

k8s-bot commented Mar 5, 2016

GCE e2e build/test passed for commit d416b56a5f594be6ae7da1948cb211d70cb8028d.

wojtek-t
wojtek-t reviewed Mar 7, 2016
View reviewed changes
docs/proposals/protobuf.md
2x speed up vs the 10x observed for Protobuf
* gRPC was considered, but is a larger change that requires more core
refactoring. This approach does not eliminate the possibility of switching
to gRPC in the future.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I think it will even be easier if we decide to do it in the future.

@wojtek-t wojtek-t assigned lavalamp and unassigned thockin Mar 7, 2016
@wojtek-t
Copy link
Member

wojtek-t commented Mar 7, 2016

@smarterclayton - very nice doc @smarterclayton - thanks a lot for writing that down!

@wojtek-t
Copy link
Member

wojtek-t commented Mar 7, 2016

I just added some comments, but those are mainly to ensure that I understood it correctly.

LGTM, but probably more people should look into it before merging.

@wojtek-t
Copy link
Member

wojtek-t commented Mar 9, 2016

Just one more point from me to incorporate your answer about tags to your doc.

@k8s-bot
Copy link

k8s-bot commented Mar 14, 2016

GCE e2e build/test passed for commit ebafdccb059907ec7163dfd92cc7cbd54d5d8f9a.

lavalamp
lavalamp reviewed Mar 16, 2016
View reviewed changes
docs/proposals/protobuf.md
* Protobuf objects on disk or in etcd will need to be self identifying at
rest, like JSON, in order for backwards compatibility in storage to work,
so we must add an envelope with apiVersion and kind to wrap the nested
object, and make the data format recognizable to clients.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, and encoding? So we can say e.g. compressed, encrypted, maybe even json or yaml (for emergency escape hatch).

@lavalamp
Copy link
Member

LGTM - a few very minor nits.

derekwaynecarr
derekwaynecarr reviewed Mar 16, 2016
View reviewed changes
docs/proposals/protobuf.md

## Implications

* The generated marshal code is large and will increase build times and binary
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity how much worse is build time and binary size?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About 15-25%.

Pushed updates.

On Wed, Mar 16, 2016 at 12:23 AM, Derek Carr notifications@github.com
wrote:

In docs/proposals/protobuf.md
#22600 (comment)
:

+handled using our existing API change rules. Tags cannot change within an
+API version.
+
+Generation would be done by developers and then checked into source control,
+like conversions and ugorji JSON codecs.
+
+Because protoc is not packaged well across all platforms, we will add it to
+the kube-cross Docker image and developers can use that to generate
+updated protobufs. Protobuf 3 beta is required.
+
+The generated protobuf will be checked with a verify script before merging.
+
+
+## Implications
+
+* The generated marshal code is large and will increase build times and binary

Just out of curiosity how much worse is build time and binary size?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/kubernetes/pull/22600/files/ebafdccb059907ec7163dfd92cc7cbd54d5d8f9a#r56280142

@k8s-bot
Copy link

k8s-bot commented Mar 16, 2016

GCE e2e build/test passed for commit 252b2fa9ae501353b8512c79fe82e82b85ad4e65.

@lavalamp lavalamp added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 16, 2016
@lavalamp
Copy link
Member

LGTM, but tests need attention

@k8s-github-robot
Copy link

PR changed after LGTM, removing LGTM.

@k8s-github-robot k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 16, 2016
@k8s-bot
Copy link

k8s-bot commented Mar 16, 2016

GCE e2e build/test passed for commit 2234c138dde2069f73c1807a1fac7a2d42ce7325.

@k8s-bot
Copy link

k8s-bot commented Mar 17, 2016

GCE e2e build/test passed for commit 95cf60b.

@smarterclayton
Copy link
Contributor Author

Fixed docs, reapplying lgtm

@smarterclayton smarterclayton added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 17, 2016
@k8s-github-robot
Copy link

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

@k8s-bot
Copy link

k8s-bot commented Mar 17, 2016

GCE e2e build/test passed for commit 95cf60b.

@k8s-github-robot
Copy link

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

@k8s-bot
Copy link

k8s-bot commented Mar 17, 2016

GCE e2e build/test passed for commit 95cf60b.

@k8s-github-robot
Copy link

Automatic merge from submit-queue

k8s-github-robot pushed a commit that referenced this pull request Mar 17, 2016
Merge pull request #22600 from smarterclayton/protoproposal
e3049d6 
Auto commit by PR queue bot
@k8s-github-robot k8s-github-robot merged commit e3049d6 into kubernetes:master Mar 17, 2016
xingzhou pushed a commit to xingzhou/kubernetes that referenced this pull request Dec 15, 2016
Merge pull request kubernetes#22600 from smarterclayton/protoproposal
7880137 
Auto commit by PR queue bot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@lavalamp lavalamp

Labels
kind/design Categorizes issue or PR as related to design. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
9 participants
@smarterclayton @k8s-github-robot @k8s-bot @wojtek-t @lavalamp @derekwaynecarr @k8s-teamcity-mesosphere @googlebot @thockin