Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC graduation application #300

Open
wants to merge 2 commits into
base: master
from
Open

gRPC graduation application #300

wants to merge 2 commits into from

Conversation

@srini100
Copy link

srini100 commented Sep 26, 2019

This is a proposal to consider gRPC for CNCF graduation.

@bgrant0607 bgrant0607 added this to In progress (due diligence/presentation) in TOC Project Backlog 2019 Q3 Sep 26, 2019
@caniszczyk caniszczyk added this to In progress in Initial Project Triage & Sandbox Projects Backlog via automation Sep 27, 2019
@mattklein123

This comment has been minimized.

Copy link
Contributor

mattklein123 commented Oct 8, 2019

@srini100 thanks for this proposal.

In terms of impact and production use, gRPC clearly qualifies for graduation from my perspective. I have a few questions about governance and language support however:

  1. Although I see 2 maintainers on https://github.com/grpc/grpc/blob/master/MAINTAINERS.md that don't work at Google, about 45 do work at Google. Can you describe the governance and design review process? Is it meaningfully possible for changes to gRPC core to be implemented without Google's consent? As a concrete example, recently gRPC decided to adopt Envoy's xDS protocol for lookaside load balancing. Was this proposal reviewed by the community or was it decided internally within Google?

  2. As I like to say, and I mean no direct offense by this statement, the magic behind gRPC is in large part due to the magic in proto3. Given that protobuf itself is not in a foundation and is controlled by Google, can you comment on the interplay between gRPC and protobuf as it relates to governance? Speaking on behalf of end-users, it has been difficult at times to get the gRPC and protobuf teams to acknowledge end user pain outside of Google (a good example of this lead to the creation of https://github.com/envoyproxy/protoc-gen-validate).

  3. Language support outside of the core supported languages by Google (C++/Java/Go) has traditionally been extremely haphazard. AFAICT this is the primary area in which non-Google contributions have come in. Can you comment on the current state of non-core languages, how you see the project evolving production ready support for these languages, and community involvement? This is somewhat related to (1) given that in some cases quality language support may require changes to gRPC core.

Thank you!

cc @jbeda

@amye amye added this to Due Diligence Complete in TOC Project Reviews Q4 2019 Oct 15, 2019
@amye amye moved this from Due Diligence Complete to PR in in TOC Project Reviews Q4 2019 Oct 15, 2019
@amye amye removed this from In progress (due diligence/presentation) in Initial Project Triage & Sandbox Projects Backlog Oct 15, 2019
@srini100

This comment has been minimized.

Copy link
Author

srini100 commented Oct 23, 2019

@mattklein123, thanks for your review and questions. Please see my responses below:

  1. Although I see 2 maintainers on https://github.com/grpc/grpc/blob/master/MAINTAINERS.md that don't work at Google, about 45 do work at Google. Can you describe the governance and design review process? Is it meaningfully possible for changes to gRPC core to be implemented without Google's consent? As a concrete example, recently gRPC decided to adopt Envoy's xDS protocol for lookaside load balancing. Was this proposal reviewed by the community or was it decided internally within Google?

gRPC governance process is described here. This was recently updated to have a well defined process for anyone to become a maintainer if they meet the described criteria (similar to other OSS projects). We are already making use of this new process to consider the request from a contributor from Skyscanner to become a maintainer of gRPC Core repo. We hope this will encourage more gRPC users to consider becoming maintainers. Over the years, many large contributions from different companies have been accepted.

The design and feature proposals are done on Github via the gRFC process which has been in place since Jan’17. Over 80 proposals have been made since then. Sometimes the initial ask may come from a user on another forum and one of the maintainers will work with them and make a proposal. Some examples of input from non-Google contributors are Service Config in DNS, C++ wrapping, custom transports and Interceptors in C#, NodeJS and Ruby. Most recently, Skyscanner, Dropbox and Google have been collaborating on Python Asyncio gRFC. This is a good example where a new set of APIs and features in gRPC Core repo are being driven by companies other than Google. Some PRs (1, 2, 3) are already merged. Other example PRs are public resolver API in gRPC Core and SSL session caching. So yes, if a non-Google contributor shows expertise, motivation and commitment and there are other non-Google reviewers with sufficient expertise, Google consent is not required. But, given the large number of committers from Google, it is natural that someone from Google will help with the reviews and influence the design and implementation.

Regarding the adoption of xDS APIs, the intention to move to xDS was announced on grpcio and in a gRFC proposal. The decision was based on user friction with the existing gRPC LB solution that was suffering from being proprietary and lacking open source control plane implementation. This enables a better ecosystem play for gRPC with other CNCF and open source projects and reduces dependence on Google for driving the roadmap.

  1. As I like to say, and I mean no direct offense by this statement, the magic behind gRPC is in large part due to the magic in proto3. Given that protobuf itself is not in a foundation and is controlled by Google, can you comment on the interplay between gRPC and protobuf as it relates to governance? Speaking on behalf of end-users, it has been difficult at times to get the gRPC and protobuf teams to acknowledge end user pain outside of Google (a good example of this lead to the creation of https://github.com/envoyproxy/protoc-gen-validate).

It is well known that gRPC and protobuf go together very well but gRPC is designed to work with other IDLs and serialization formats and similarly protobuf can be used with other communication protocols too. We take extra care in not taking a dependency on protobuf in gRPC core (example) and keeping gRPC independent of protobuf. We published a blog to show how easy it is to use JSON with gRPC. Microsoft has also published their serialization protocol Bond over gRPC.

Regarding the governance, these are treated as independent projects. Committers/maintainers are different and have good cordial relationship. Just like gRPC has interaction with Netty and now with Envoy Data Plane APIs, it has interaction with Protocol Buffers. Most of the user friction with protobuf is unfortunately around protobuf tooling which is outside the scope of gRPC project. gRPC & Protobuf teams acknowledge the user pain. Protobuf related tools is a great place to start community contributions. It is orthogonal to gRPC project, but as a part of Protobuf community efforts, the idea of starting community efforts to improve protobuf tooling is very appealing.

  1. Language support outside of the core supported languages by Google (C++/Java/Go) has traditionally been extremely haphazard. AFAICT this is the primary area in which non-Google contributions have come in. Can you comment on the current state of non-core languages, how you see the project evolving production ready support for these languages, and community involvement? This is somewhat related to (1) given that in some cases quality language support may require changes to gRPC core.

Besides C++, Java and Go, Google maintainers are also fully supporting Python, Ruby, Node, ObjectiveC, C# and PHP. All these languages are on par with core languages in terms of features and version releases. This shows that Google is committed to supporting more languages than just the core languages, but of course, Google maintainers themselves cannot develop gRPC for a number of other languages that the community is interested in. To have gRPC available in more languages, we rely on community contributions which do not require Google consent. For example, gRPC-Swift is being jointly developed by Apple, Lyft, Timing and Google. Currently, it relies on gRPC C-core but soon it will move to Swift-NIO based stack for production readiness. gRPC-Dotnet is another example where contributions from Microsoft has enabled production use of gRPC in ASP.NET Core. This was announced in .NET Conf recently. Even for existing languages, we are happy to accept large community contributions like the Python Ayncio I mentioned above and Interceptors in C#, NodeJS and Ruby. For a language support to go GA we will ensure that it passes interop tests with other implementations and a sustainable test infrastructure is available for long term support.

Thanks!

@srini100

This comment has been minimized.

Copy link
Author

srini100 commented Oct 24, 2019

Updated the "security audit" section with details of a third-party security audit that was recently completed.
Also, added Skyscanner as one of the maintainers of gRPC Core repo.

@caniszczyk

This comment has been minimized.

Copy link
Contributor

caniszczyk commented Nov 18, 2019

Who wants to volunteer from the @cncf/toc to do the due diligence for gRPC?

@bgrant0607 ?

@ultrasaurus

This comment has been minimized.

Copy link
Member

ultrasaurus commented Nov 20, 2019

Should due diligence be prepared by someone who doesn't work for Google? There may be a perceived conflict of interest.

Perhaps a TOC Contributor could draft due dilligence or someone from the group who part of proposed SIG-Networking (see PR #317)

@xiang90

This comment has been minimized.

Copy link
Contributor

xiang90 commented Nov 22, 2019

happy to help if TOC from Google is a concern.

@thisisnotapril

This comment has been minimized.

Copy link

thisisnotapril commented Jan 7, 2020

hi all; would love to get this moving - any update?

@amye amye added this to In progress (due diligence/presentation) in Initial Project Triage & Sandbox Projects Backlog Jan 8, 2020
@xiang90

This comment has been minimized.

Copy link
Contributor

xiang90 commented Jan 10, 2020

hi all; would love to get this moving - any update?

I am happy to help on the DD. Usually the project owners + the CNCF SIG will make a draft. And the TOC member will give it a review, and "approve" it. If everything is OK, the voting process will begin.

The DD template is here: https://github.com/cncf/toc/blob/master/process/dd-review-template.md

Here are some recent DD examples:

  1. Falco: https://docs.google.com/document/d/1TJCzW8dQ6858lw2UNY-H5LMnvEd4GzwjuOcDInimeyA/edit?usp=sharing

  2. Harbor: https://docs.google.com/document/d/15gX7EeeXQThEvVMGpL-0a1mOwGuByLtMfvXNJaKT0A0/edit?usp=sharing

  3. Vitess: https://docs.google.com/document/d/1TDlRdgfTiEWunpav-G8gkaQF7Zk84-9tNAXyv1I0Kws/edit?usp=sharing

@amye amye added this to Needs TOC Triage & Public Comment Kickoff in Graduating Projects Backlog Mar 13, 2020
@michelleN michelleN removed this from In progress (due diligence/presentation) in Initial Project Triage & Sandbox Projects Backlog Mar 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Graduating Projects Backlog
Needs TOC Triage & Public Comment...
TOC Project Backlog 2019 Q3
  
In progress (due diligence/presentation)
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants
You can’t perform that action at this time.