Unclear how to specify upstream (webserver) HTTP protocol

[evankanderson]

TL;DR: I need to be able to specify whether application pods behind an HTTPRoute support HTTP2 over cleartext (prior knowledge or h2c) and/or websocket. In particular, my pods do support both, but the Contour implementation (at least) defaults to vanilla HTTP/1.1, and the API spec as a whole doesn't seem to provide any way to change that.

Background:

I'm attempting to fix up Knative's Gateway-API adapter, and I've run into a problem getting some of our tests (which work with both Istio and Contour using their native interfaces) to work with Gateway-API implementations. In net-contour, we have the following bits, which I believe roughly correspond to Envoy options:

https://github.com/knative-sandbox/net-contour/blob/main/pkg/reconciler/contour/resources/httpproxy.go#L216-L224

I'm not sure how you want to do this; @youngnick suggested using the AppProtocol field of v1.Service, but I believe all the variations mentioned above are valid on port 80 / www / http / www-http, so it's not clear how you'd toggle the "supports Websocket" bit, or the "Upgrade to HTTP/2" bit.

[youngnick]

Yeah, I seem to recall us discussing using the AppProtocol field of v1.Service, which can approximate the way that Contour uses a protocol field in HTTPProxy route targets to select if the backend is tls, h2, or h2c (those are the only supported options).

It's intended that the front protocol be represented with the Listener protocol field, so the "Upgrade to HTTP/2" would be handled by setting the Listener to HTTP or, more likely, HTTPS, and then the AppProtocol on the Service to h2c, I think. (I don't know if any of this is actually implemented anywhere, but this is how it seems like it should work to me).

None of that covers "supports Websockets" though. In HTTPProxy, we have a separate field for that, along with an annotation that could go on an Ingress object. (Obviously annotations are to be avoided here if possible, but worth noting for historical reasons).

So, in summary, I think that the combination of the Listener protocol and the AppProtocol should be able handle the "Upgrade to HTTP/2" case, but that we need to figure out what to do for the "Supports Websockets" dimension (which is not entirely orthogonal, but may be better served with a separate mechanism.)

[howardjohn]

you can passthrough TLS in TCProute without actually knowing it's TLS as well

[robscott]

you can passthrough TLS in TCProute without actually knowing it's TLS as well

Yep, fair point. I think what I was getting at is that any kind of appProtocol guidance here only seems relevant to HTTPRoute.

[Miciah]

I agree that appProtocol solves only a narrow set of use-cases, and addressing that set of use-cases using appProtocol could reduce motivation to design a broader solution. Still, I believe it is worth documenting some kind of expectations around appProtocol. Please consider that even once we have a broader solution, appProtocol will still exist, so it will still make sense even then to set some expectations. And as long as we're setting expectations, we might as well set reasonable expectations that solve some needs (grin).

[Miciah]

Granted, the right place to set those expectations might be in the Service API (which I believe you are suggesting in another thread on this discussion). Then I would suggest Gateway API cite the relevant Service API documentation once it exists.

[evankanderson]

I think an interesting question is whether appProtocol is actually fit for purpose.

The constraint on reserved IANA names makes sense on its face, but given the actual evolution of the internet (layers of "backwards compatible" protocols, especially on HTTP + TCP/UDP) it's not clear that "port number assignment" means as much as it did 15 years ago.

[bowei]

Is there a discovery protocol for HTTP2 (2c in particular).

@robscott was there some standardization on the format for AppProtocol wrt to to HTTP/2?

[evankanderson]

I have some (old) code that speaks HTTP/1.1, HTTP/2, and Websockets in a single binary:

https://github.com/evankanderson/sia

I'd be willing to modernize it and contribute it to the conformance tests if we can agree how it would work to pass this through. There's also the question of how to up- or down-convert compatible requests if (for example) a browser wants to speak HTTP/2 across the internet (possibly with TLS), but an upstream only speaks HTTP/1.1. It should be possibly for the Gateway to do that conversion; there are two possibilities that I see:

1. The gateway tracks (maybe on a per-upstream basis) the capabilities of that upstream, using the HTTP upgrade capabilities to select the best mode for a given request. As an end-user, I sort of like this, because I have zero work to do, it's all up to the Gateway to get it right. But we need to make that a conformance requirement, or I end up with a bunch of different solutions, including multiple different solution 2s.

   My understanding is also that Envoy doesn't currently implement this option, which may steer us toward option 2.

2. Provide an external configuration option indicating what protocol(s) the upstream is able to speak, and then the gateway can rely on static information to determine the best path for a given request when communicating with the upstream. I think this is what Envoy supports today (or at least, what it supported 2 1/2 years ago when I last read the documentation in detail), and is where the suggestion of using something like AppProtocol comes in.

The worst scenario as a consumer of Gateway API is that each implementation ends up documenting their own version of 2, and I as a Knative implementer end up with 5 different implementation-specific annotations on each Service I create, plus specific Name and AppProtocol values I need to set to hint to the implementations what protocol to use for connection. (I suppose the very worst would be that I have to add a flag to say "speak $IMPLEMENTATION_A port Names" vs "speak $IMPLEMENTATION_B port Names, but I don't want to think about that...)

[robscott]

The gateway tracks (maybe on a per-upstream basis) the capabilities of that upstream, using the HTTP upgrade capabilities to select the best mode for a given request.

I agree that this seems like a nice option but also is unlikely to be universally implementable, so we're stuck with providing some kind of configuration here.

Provide an external configuration option indicating what protocol(s) the upstream is able to speak, and then the gateway can rely on static information to determine the best path for a given request when communicating with the upstream.

Yeah, agreed that we need something here. I think this lines up with @bowei's question above around if AppProtocol has been standardized here. It links out to https://www.iana.org/assignments/service-names, which is a long list of service names but it misses very common concepts like grpc, h2, and h2c. We can solve that specific problem with a PR to k/k that defines names to use for some of the most common cases like this.

Unfortunately that only solves part of the problem here. The other part is when we want to communicate that an upstream can support multiple protocols. I'm not sure what the best way to handle that is, but open to ideas.

The worst scenario as a consumer of Gateway API is that each implementation ends up documenting their own version of 2, and I as a Knative implementer end up with 5 different implementation-specific annotations on each Service I create, plus specific Name and AppProtocol values I need to set to hint to the implementations what protocol to use for connection.

💯

[evankanderson]

Unfortunately that only solves part of the problem here. The other part is when we want to communicate that an upstream can support multiple protocols. I'm not sure what the best way to handle that is, but open to ideas.

It's not clear whether grpc, h2 and https are different protocols.

This may be a case where port numbers are insufficient for describing the capabilities within a single protocol (HTTP in particular -- IPP, WebSocket, and HTTP/2 are all extensions of HTTP).

[evankanderson]

One terrible solution would be to allow communicating multiple protocol extension capabilities in AppProtocol with +.

[bowei]

It sounds like something User-Agent ish (Yes, I know how awful user-agent is :-)

[youngnick]

Reading back over this, I thought it might be worth writing down what I think the problem is, and what I think we could do about it. (We had a similar problem in Contour for HTTPProxy, so there's some relevant experience to add.)

We need to have a way to be able to supply details for backends about:

• what protocols they support (TLS, gRPC, Websockets, other stuff in the future)
• Details you need to use those protocols (TLS certs, Websocket paths, probably more things too)

These are mostly distinct from the Listener-side behavior for the HTTPRoute case, and are pretty limited for the TLSRoute case. They shouldn't matter for the TCPRoute or UDPRoute case.

They'll also be necessary for some uses of the new GRPCRoute when it's done, I guess.

Seems to me like we have a few options:

Use Service's spec.ports[].appProtocol for determining this info.

Pro:

• Already existing, no changes required.
• Puts the constructs at the level that they'll be useful in other contexts, like Mesh/GAMMA ones.
• Putting everything into the Service means there's only one place, which means there's no conflict resolution/override hierarchy design necessary.

Cons:

• Can't meet extra data requirements, we'd need to add something to BackendRefs (probably) anyway.
• AppProtocol is limited to service names defined in the Service Name and Transport Protocol Port Number Registry, for anything else we would need to use domain-prefixed names. Doable, but doesn't seem ideal.
• If we want to change things in Service, changing a v1 API takes a long time (at least 9-12 months).

Add something to to the Gateway API

This would cover adding extra details on a per-backend or per-Route basis.

Pro:

• Much quicker turnaround time, we're adding stuff so it can go straight into v1beta1 resources, via the experimental channel.
• We can add whatever we like, so storing extra info is reasonably straightforward.
• Contour did something very similar, so there's at least some prior art (I haven't checked anyone else's CRDs yet though, but I'd be surprised if most of them didn't include something).

Con:

• If we put this in the xRoute resources, it's further away from the endpoint grouping construct (that is, the Service in most cases). This will probably have impacts on how the mesh/GAMMA constructs are added.
• Doing things here means we need to define override behavior and how AppProtocol interacts with these fields.

Use a wrapper/meta resource

This would cover adding a ServiceMetadata resource or something, that has a reference to a Service, and the extra fields we would otherwise add.

Pro:

• Control of the resource, so we can go faster, similarly to the second option.
• We can add what we like, so storing extra info is easy.
• A meta-resource keeps all this info in the one place, so we should have reduced need for conflict resolution.
• The GAMMA initiative has been talking about something similar for mesh use cases as well.

Con:

• Another resource to manage, reconcile and design around.
• We don't completely remove the need for conflict resolution, we'll still need to define what AppProtocol does in this case.

This turned into an essay, but maybe people could have a think about which path they prefer. I'm not fully across the various Mesh design options yet (That's on the list for tomorrow) or early next week, and maybe some folk who are more familiar there could speak about the overlap.

I guess I should also say that those aren't hard options or anything, just stuff off the top of my head. We could do one, more than one, or none of those things.

[Miciah]

What about using policy attachment to attach a policy to a service? Is that outside the remit of policy attachment? The linked documentation does note that there would be no way to update the status of a service object to report that a policy had been attached, but if that tradeoff is acceptable, then I don't see anything that would rule out the approach.

Edit: https://gateway-api.sigs.k8s.io/references/policy-attachment/ is a better documentation link (images don't show up with the other link).

[keithmattix]

Hmm that's not a bad idea actually...a kind of baked in service policy attachment resource that could be composed with implementation specific resources 🤔 It does get difficult to figure out "what are all of the policies applied to this service" if you can't check the status though. The more I think about it, I think I still would prefer a separate wrapper resource that policy attachments would bind to instead, the status control being one big reason why.

[evankanderson]

The Service/Endpoint duality, DNS mapping, and the different (pseudo-concentric) types of Services are definitely a lot of magic to unpack. I don't want to make that worse-r than we have to.

Unfortunately, at the end of the day, unless we define a new v2/Service interface, that bundle of magic is the common lingua franca that every network-speaking component needs to deal with in Kubernetes. It feels more useful to improve that base layer. In particular, I'd hate to get to a Service/ServiceMetadata/Endopint/EndpointSlices quadrifecta for "common" use cases -- Service and label matching is enough complication for people getting started with Kubernetes.

[keithmattix]

That's a good point, and I agree; we should strive to make the common case easy. However, I feel like appProtocol is the common case, and if users want to do something like handle layers of TLS certificates, identity federation, etc. then that's a bit past the realm of common use-cases. I'm not saying those things shouldn't be relatively simple to achieve (truthfully, I'd love to see cloud providers/platforms build abstractions on top of these things), but as an spec, I feel like Gateway API should lean towards expressing a concept completely and correctly. It may very well be that v2/Service comes along and borrows a lot of implementation details from a ServiceMetadata resource, but I don't get the sense that Kubernetes wants to tackle that at the moment. I could be wrong there, but I think we probably have a better chance of testing and proving a specific model works in a more limited setting (i.e. here in Gateway API) before trying to muster up support for a KEP, leaving users with vague guidance at best in the meantime.

[youngnick]

I think we should in general be careful to use Policy Attachment for clear settings that make use of the defaults and overrides concepts. I think that the problem we're talking about here is a bit larger than I'd like to see that mechanism used for.

I favor the ServiceMetaData or similar approach for adding extra information onto a Service. The behavior I have in mind would either be that you can specify the ServiceMetadata thing directly, as a backend, or that it gets used if it exists (which feels a little too magic to me).

I don't disagree that Service should have this sort of info in it, but I think the timelines and effort required make altering or bumping the version of Service too costly. I'm also -1 on using annotations on Service, because then we're stuck using those annotations for a long time.

I'd much rather have the flow be:

• Service for super simple use cases.
• If you want more complexity, you have to use ServiceMetadata resource. This gives us the space to slowly migrate things out of ServiceMetadata into Service if it's all too onerous.

The last reason for this is that GAMMA use cases are going to need basically the same thing anyway, and this lets us only do this once.

I'm also going to throw open the door to chaos by suggesting that the resource should be called ServiceConfig ExtendedService or some other name that indicates its wrapper nature.

[youngnick]

I've opened GEP #1282 to track the overall work to sort out a full solution to this. I'll write up the start of what I think it should look like in a gdoc and post the link in there.