KEP-3726: Add 'kubernetes.io/raw' as a standard app protocol

[dprotaso]

This PR is a continuation of #3996 - simplified diff is here 93d0269

• One-line PR description: This defines kubernetes.io/raw as a standard app protocol

• Issue link: Standard Application Protocols #3726

This is completing the follow up work outlined in the KEP-3726 update introduce here - #3912

Specifically

Some implementations are trying to guess the application protocol in absence of appProtocol. We should consider adding a new protocol like kubernetes.io/raw to the collection that would instructs implementations to process requests as raw. We need to look at combination of appProtocol: kubernetes.io/raw and the different supported port protocols (TCP, UDP, SCTP).

Motivation

Gateway API GEP - kubernetes-sigs/gateway-api#1911

Problem: How can users specify their traffic to a service/endpointslice port is exactly what's specified in protocol and automatic detection should be turned off.

I saw that the standard protocol KEP had a clause about raw - but it wasn't added.

[shaneutt]

For posterity, could we please get more of the context pulled in from the related bits into the PR description here?

[dprotaso]

/assign @shaneutt @robscott @thockin

[dprotaso]

For posterity, could we please get more of the context pulled in from the related bits into the PR description here?

Done

[thockin]

Can we say what is the motivating case for this?

[thockin]

Traffic to any port is the protocol defined by protocol. That's tautological. How about something like:

"Opaque traffic which should not be handled inspected further"

[sftim]

That's not quite right either though @thockin. We still want the node OS (or container runtime sandbox, if it's something like gVisor) to process the lower layer stuff. But there's no application protocol on top.

I don't want anyone to misconstrue this as an opt-out from their security appliance and packet / flow inspection.

How about

Suggested change

	// * 'kubernetes.io/raw' - Traffic to this port is the protocol value defined by this port's `protocol` field. (ie. TCP/UDP/SCTP)
	// * 'kubernetes.io/raw' - Opaque application traffic; implementations should not apply special processing

[dprotaso]

ok updated. PTAL

[dprotaso]

Can we say what is the motivating case for this?

Gateway API GEP - kubernetes-sigs/gateway-api#1911

How can users specify their traffic to a service/endpointslice port is exactly what's specified in protocol and automatic detection should be turned off.

Then I saw that the standard protocol KEP had a clause about raw

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dprotaso
Once this PR has been reviewed and has the lgtm label, please ask for approval from shaneutt. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-network/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[thockin]

I meant we should add text here explaining that sometimes implementations assume they can "sniff" the protocol and sometimes that causes problems and we don't want them to.

What I am struggling with is why this is generic API and not "hey <impl>, don't be stupid" ? Why would a user WANT to specify "raw" that doesn't boil down to "I know something about my impl that needs me to say work around it"?

[sftim]

Maybe kubernetes.io/opaque is a better name?

You might want to validate at admission time that an app protocol has been declared I guess. Not sure how a null value here is different from opaque though.

[dprotaso]

disclosure: not an ingress implementator

For the Gateway GEP I could suggest that if the appProtocol is not set then assume the target will accept traffic as defined by protocol and don't perform sniffing. But my guess is most users do not set appProtocol so there is a advantage to sniffing.

Thus having some constant to suggest that protocol should be interpreted as absolute is what I'm trying to do.

If we feel that this doesn't belong as a k8s standard protocol I can bring this to the Gateway folks for a discussion.

[howardjohn]

FWIW as an implementation that does sniffing (but not the first - Linkerd was as far as I know), the reason we added it was our onboarding guide was something like: 1. Install service mesh 2. Go update all your port definitions, in every service, including third party helm charts, etc 3. Enjoy service mesh-y features Step (2) was a big big hurdle to adoption.

…

On Wed, Jul 12, 2023 at 9:39 AM Dave Protasowski ***@***.***> wrote: disclosure: not an ingress implementator For the Gateway GEP *I could* suggest that if the appProtocol is not set then assume target will accept traffic of as defined by protocol and don't perform sniffing. But my guess is most users do not set appProtocol so there is an advantage to some implementations performing this sniffing. Thus having some constant to suggest that protocol should be interpreted as absolute is what I'm trying to do. If we feel that this doesn't belong as a standard protocol I can bring this to the Gateway folks for a discussion. — Reply to this email directly, view it on GitHub <#4106 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXP7ASGCW4Q3K423Z6LXP3HMXANCNFSM6AAAAAAZMCXY6A> . You are receiving this because you commented.Message ID: ***@***.***>

[thockin]

What I am asking is - why is sniffing ever bad? Is that implementation-centric or actually fundamental? It feels like "raw" is a way to express "I know I am using istio, and I know istio is broken in this specific way, so let's disable that".

[howardjohn]

What I am asking is - why is sniffing ever bad? Is that implementation-centric or actually fundamental? It feels like "raw" is a way to express "I know I am using istio, and I know istio is broken in this specific way, so let's disable that".

Ah, good question. The main issue is that for some protocols (mysql, most commonly), its entirely broken: https://istio.io/latest/docs/ops/deployment/requirements/#server-first-protocols.

There are also some more niche use cases where you already know its not HTTP so you why bother with the complexity, but I think that is mostly hypothetical

[thockin]

So, in this case it really is "I know something about my implementation". But why do we want "raw" instead of "mysql" ?

[howardjohn]

So, in this case it really is "I know something about my implementation". But why do we want "raw" instead of "mysql" ?

We could, but mysql is just one common case. I have seen maybe 10 different protocols that are "server first" in my experience, and I am sure there are more - so it can be challenging to keep up with.

[thockin]

I have seen maybe 10 different protocols that are "server first" in my experience, and I am sure there are more - so it can be challenging to keep up with.

So assume I am a user who invents a new, bespoke protocol which is server-first. Istio obviously doesn't support me. If I am using istio and sniffing breaks, should I specify kubernetes.io/raw or istio.io/dont-sniff ? Whats the value in a "common" name for this? I don't think we expect people to set it by default on every port, right?

[howardjohn]

So assume I am a user who invents a new, bespoke protocol which is server-first. Istio obviously doesn't support me. If I am using istio and sniffing breaks, should I specify kubernetes.io/raw or istio.io/dont-sniff ? Whats the value in a "common" name for this? I don't think we expect people to set it by default on every port, right?

You could and that would work fine, but its not really portable which seems to be the goal of all of the protocol definitions, right?

You could do istio.io/http, but there is value in telling everyone its "HTTP".

Similarly for raw, I may want to tell linkerd, kuma, etc as well. A single user may not really care (you probably only use one mesh), but a common helm chart may not want to use a vendor specific label for example.

Granted, I could make the same argument about almost any appProtocol value, and its a bit of a slippery slope...

[thockin]

Similarly for raw, I may want to tell linkerd, kuma, etc as well.

If we define "standard" raw, and then some implementations get better at sniffing, we've created an attractive nuisance. Premature pessimization.

"raw" is different from the rest - the others are affirmative statements, "raw" is not telling us what it IS, it's a bypass. That's why I am unsure

[dprotaso]

"raw" is not telling us what it IS,

My original definition was that "raw" meant look at the protocol field and don't assume anything else. That to me is affirmative and by-passing sniffing is a side-effect for some implementations.

I think an alternative would be to specify IANA names for TCP, UDP, etc in the appProtocol field - but are we ok with the duplicate values for protocol and appProtocol?. This question might be more for the Gateway GEP and not this KEP.

[aojea]

raw does not look something as a standard app protocol honestly, why not empty means raw?

[sftim]

raw does not look something as a standard app protocol honestly, why not empty means raw?

The challenge is to tell apart empty-by-intent from empty-by-default. Lots of clusters will have objects with no value specified for this field.

[thockin]

My original definition was that "raw" meant look at the protocol field and don't assume anything else

The absence of an AppProtocol really means "I have nothing further to express here". I should not need to explicitly say "this page intentionally left blank". It's implementations which are choosing to try to be clever (and sometimes failing). The desire to express "raw" derives from implementation(s) which fail to cover their tracks.

We're telling the end user "it's not your fault, but it is your problem -- you have to take action to clean up our mess". Yuck.

I'm not against "raw", I just think it shouldn't be "kubernetes.io/raw". If mesh implementations want to agree on a common name, that's outside my purvey.

[sftim]

I'm inferring that mesh vendors might agree on mesh-consortium.example/opaque and document the bodge to their users.

[aojea]

We're telling the end user "it's not your fault, but it is your problem -- you have to take action to clean up our mess". Yuck.

sorry, I have to make the joke 🥲 kubernetes.io/mess

[thockin]

kubernetes.io/do-what-i-mean-not-what-i-said

…

On Thu, Jul 13, 2023 at 1:27 AM Antonio Ojea ***@***.***> wrote: We're telling the end user "it's not your fault, but it is your problem -- you have to take action to clean up our mess". Yuck. sorry, I have to make the joke 🥲 kubernetes.io/mess — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[thockin]

So after all this discussion, I think we say thanks, but no thanks to this one, right?

[dprotaso]

Sounds good