Define Initial APIs

[danehans]

#16 specifies the need to define APIs, e.g. bootstrap config. To begin developing these APIs, the community should agree on:

1. How these APIs will defined, protos vs Go types. Emissary uses protos to define API types whereas Contour uses Go types.
2. How these APIs will documented.
3. How these APIs will be maintained.
4. Others?

[danehans]

xref config api: #29
xref internal api: #32

[arkodg]

+1 for defining APIs using protos

• acts as source of truth for doc strings
• schema can be used to validate fields https://github.com/envoyproxy/protoc-gen-validate
• keeps this project consistent with envoy https://github.com/envoyproxy/envoy/tree/main/api

[danehans]

@skriss @youngnick I'm interested in your feedback here since Contour does not use protos. It would be great to achieve consensus so we can start building out the APIs.

[danehans]

xref Bootstrap API: #51

[danehans]

xref @arkodg PR for bootstrapping proto-based API tooling: #53

[arkodg]

the decision was made in the maintainers meeting to use Kubebuilder based APIs over Protobufs

• @LukeShu @youngnick I couldn't find any links for tooling inside kubebuilder that tracks API compatibility & breaking API changes, can you please share any links in this area. For e.g. buf has a subcommand to track breaking API changes.

[danehans]

@LukeShu per the 5/24/22 meeting notes, please document the rationale for using kube-builder/controller-gen over protobuf.

[LukeShu]

Sorry, I didn't get to this on Wednesday, and have been on vacation since then. Will get that documentation up soon.

[LukeShu]

Background

For those outside of the know: The kubebuilder CLI is essentially a scaffolding-generator; a key part of what it does is generate you a Makefile that calls the controller-gen program when you run make generate. When people say "using kubebuilder", they mostly mean "using controller-gen".

Rationale

So why use the kubebuilder ecosystem instead of the Protobuf ecosystem, which Envoy has invested heavily in?

• We're defining extensions to the Gateway API. The Gateway API is canonically specified using controller-gen; using the same tooling means we don't have to learn two sets of tooling (assuming that those involved with Envoy Gateway will also have some level of involvement with Gateway API).

• Protobuf does have an official JSON representation, however this representation is sloppy about whether capitalization matters in key names. This is at odds with Kubernetes, where the capitalization does matter. Protobuf is opinionated about the capitalization of enum values, but also permits numeric representation of those values; the int/string-duality isn't compatible for Kubernetes structural schemas . In the past, parts of Emissary's API were specified via Protobuf, and this was a continual source of headaches. Perhaps this is solvable with newer/better tools.

• The validation language (magic // +kubebuilder: comments in Go source) of controller-gen exactly matches the validation capabilities of the Kubernetes apiserver and kubectl client. For UX reasons, it is desirable to shift validation "left" as much as possible. There are tools that translate protoc-gen-validate annotations in to validations that the apiserver and kubectl can consume, but ultimately they are being consumed by a different validation machine than protoc-gen-validate is designed around.

• Within the Kubernetes ecosystem, for new projects, Kubebuilder is absolutely the "default" choice. At one point some of the built-in Kubernetes APIs were canonically specified using Protobuf, but have since migrated to be canonically defined as Go types (using an internal predecessor to conversion-gen). For various reasons, for better or for worse, the Kubernetes ecosystem is moving away from protobuf for specifying resources.

Perhaps the pain points with Protobuf in this use-case are solvable with newer/better tools (there's one @arkodg mentioned that I hadn't heard of before, and whose name is escaping me at the moment). But ultimately, we're specifying a Kubernetes API, and it makes more sense to use a tooling ecosystem that is aligned with that. Let's go with the mainstream choice; and Kubebuilder absolutely is the "default" choice for new Kubernetes projects.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[danehans]

#51 defines the initial config APIs for control and data planes.