KEP: Adding AppProtocol to Services and Endpoints

[robscott]

The lack of standardized application protocol support for Services and Endpoints has frustrated countless Kubernetes end users. With EndpointSlices, we added a new AppProtocol field as a first step, adding that same field to Services and Endpoints will ensure full API support for application protocols.

This issue was initially raised here where @thockin suggested filing a KEP. I'm happy to also file a PR with implementation if this is something we want to proceed with. I'd love to get this into 1.18 if possible.

/sig network
/cc @thockin @dcbw

[robscott]

/assign @thockin

[thockin]

I'm fine with this, but I want you to do 2 things:

1. Let this cook for a week or two past the holidays, and socialize it at the next sig-net
2. Reach out to folks who own the existing annotations and see if we can get them to support this

/approve
/lgtm
/hold

You can remove the hold when the above is done ;)

[lachie83]

cc @feiskyer @andyzhangx @khenidak @aramase - can you please review?

[M00nF1sh]

I'm good with this since it's ready included in EndpointSlice object.(And can help add support for this from AWS side)

Only concern is that the root problem(in my understanding) is we are lacking support to configure application-specific hints on per port basis. e.g.HealthCheckProtocol/HealthCheckInterval/SSLCertificate/etc, so this will only solves a subnet of the problem.

/lgtm

[thockin]

Your "concern" sounds like a flat statement more than a problem, to me. Can you expand on that?

…

On Fri, Jan 3, 2020, 6:42 PM M00nF1sh ***@***.***> wrote: I'm good with this since it's ready included in EndpointSlice object. Only concern is that the root problem(in my understanding) is we are lacking support to configure application-specific hints on per port basic. e.g. for cases where customers want to use different TLS cert per port. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1422?email_source=notifications&email_token=ABKWAVER2JVFRPVOZ5DH34DQ37ZP5A5CNFSM4KADTDMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEICPH6Y#issuecomment-570749947>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVCJPHGDLQV6SHT7GXDQ37ZP5ANCNFSM4KADTDMA> .

[feiskyer]

Looks good in general, but I'm agreeing with @M00nF1sh that even with this new AppProtocol, other annotations may still be required to fully support the feature.

Take https for example, SSLCertificate is also required to make it work. So another annotation is still required to use the https protocol.

Do we consider to support more protocol details (e.g. application options) together with this new property?

[thockin]

The struggle here is to stay simple and neutral to implementations while providing enough hints that most users can express what they need. I am not inherently AGAINST further extension, but it would have to be carefully considered.

…

On Mon, Jan 6, 2020, 6:26 AM Pengfei Ni ***@***.***> wrote: ***@***.**** commented on this pull request. Looks good in general, but I'm agreeing with @M00nF1sh <https://github.com/M00nF1sh> that even with this new AppProtocol, other annotations may still be required to fully support the feature. Take https for example, SSLCertificate is also required to make it work. So another annotation is still required to use the https protocol. Do we consider to support more protocol details (e.g. application options) together with this new property? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1422?email_source=notifications&email_token=ABKWAVFKG6IKMPO2KSFYNFDQ4M5STA5CNFSM4KADTDMKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCQX3LNY#pullrequestreview-338671031>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVBATIOCFGU2UWRO7WDQ4M5STANCNFSM4KADTDMA> .

[khenidak]

Thanks @robscott -

Let us talk about this in sig-network. I fail to see the value that justifies the complexities that this will bring. In a lot of ways, this moves services from objects that describes L4 to objects that describe L4 and L7

[thockin]

Kal, This is a "hint" for consumers who want to do something L7ish on top of L4. Others can ignore it. I don't think it makes Service into L7 (though that is a slippery slope). The reality is that many implementations are already carrying this data via annotations or (puke) port names (I'm looking at you, Istio). I proposed to formalize this as a way to get the most basic primitive extracted. That said, I recognize the slippery slope, and I have been on the other side of this argument in the (distant) past.

…

On Mon, Jan 6, 2020 at 9:08 AM Khaled Henidak (Kal) < ***@***.***> wrote: Thanks @robscott <https://github.com/robscott> - Let us talk about this in sig-network. I fail to see the value that justifies the complexities that this will bring. In a lot of ways, this moves services from objects that describes L4 to objects that describe L4 and L7 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1422?email_source=notifications&email_token=ABKWAVASCW7QCPPH6OZQJW3Q4NQRBA5CNFSM4KADTDMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIGCYFQ#issuecomment-571223062>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVCW3J24KSNGA3CSGATQ4NQRBANCNFSM4KADTDMA> .

[mattmoor]

@thockin it would be useful to be able to support such a hint in the PodSpec itself as well.

[thockin]

@mattmoor Can you talk about the use case distinct from Service?

[mattmoor]

It's effectively the same as for a Service, but there are use cases (e.g. Knative) where we have a need to let the user specify the application protocol where they can provide a PodSpec (incl. ports), but we synthesize the K8s Service and related Ingress abstractions around that.

Currently we have the pleasure of following in the Istio footsteps you are so fond of, but would love something better. 🤣

The simplest example of where a PodSpec level hint would help is: How would this hint get populated on Services created via kubectl expose? 🤷‍♂

[hbagdi]

This would help the Ingress use-case quite a bit!

Questions:

• How would this work with #1438?
  There are usually multiple protocols above L4 that are possible on the same port via ALPN, upgrade headers or what not. We often expose gRPC, HTTP/2, HTTP/1.1 (all on top of TLS) on the same port (443). And in future, HTTP/3 will be added into that list.
• Will Kubernetes itself make use of this or is it only a hint for other applications and integrations? Seems like later, but then is it possible to add guidelines on who decides the protocol to be set? Imagine that my app runs gRPC, which uses HTTP/2.0 for transport. Now, a health-check service will want to treat the AppProtocol as HTTP, but a proxy will want gRPC.

[robscott]

Hey @hbagdi, thanks for the great questions! It seems like there is a fairly common desire to support multiple application protocols per port number. Under the current restraints, the most straightforward way to do that would be to transform the proposed AppProtocol field into an AppProtocols list.

Alternatively, if #1438 is approved and removes the requirement for port numbers in a list to be unique, that could provide a way to specify multiple app protocols simply by having multiple port entries.

Either way, this is not a field Kubernetes itself would use, but would be a hint for consumers of the API. Not all consumers will support all app protocols, but the field can serve as a hint for those that do.

@thockin How would you feel about supporting multiple app protocols per port?

[thockin]

@mattmoor I can perhaps buy that, but it probably warrants a small KEP of its own.

@hbagdi I think I'd like to start small and expand if we can see really concrete use cases. I admit I don't know enough about them. I'm going to look at and hopefully approve this KEP as is, but if we can flesh out why multiple app-protocols is really important, that seems minor enough not to invalidate this. Note that we would want to carry that through to EndpointSlice, which already exists in the singular form.

[thockin]

Thanks!

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: M00nF1sh, robscott, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-network/OWNERS [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[robscott]

/hold cancel