SCTP support implementation for Kubernetes

[janosi]

What this PR does / why we need it: This PR adds SCTP support to Kubernetes, including Service, Endpoint, and NetworkPolicy.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #44485

Special notes for your reviewer:

Release note:

SCTP is now supported as additional protocol (alpha) alongside TCP and UDP in Pod, Service, Endpoint, and NetworkPolicy.  

[janosi]

/assign @thockin

[thockin]

I know precious little about SCTP - I have no hands-on experience. @kubernetes/sig-network-feature-requests I do know, for example that GCP's load-balancer only supports SCTP in single-endpoint mode: https://cloud.google.com/compute/docs/load-balancing/network/forwarding-rules

So at a minimum, all of the cloud implementations need to be checked for compatibility and disabled when not compatible.

This should probably start as a proposal to sig-network to gather context (like the above GCP point) and then become a KEP for broader review.

[janosi]

@thockin Do I understand right, that you would like us to go back to point 0 and start with a sig-network proposal first, and then after some criteria are met we should create a KEP? In order to check that if a cloud provider API gets an unsupported protocol value then it answers with an error and it is handled right in the relevant cloud provider logic of k8s?

[janosi]

@thockin
I have checked now the EnsureLoadBalancer implementations of the different cloud providers (pkg/cloudprovider/providers)
AWS: it accepts only "TCP" as Protocol. Otherwise it returns with error.

kubernetes/pkg/cloudprovider/providers/aws/aws.go

Line 3249 in 52603a7

if port.Protocol != v1.ProtocolTCP {

Azure: it accepts only "TCP" or "UDP". Otherwise it returns with error.

kubernetes/pkg/cloudprovider/providers/azure/azure_standard.go

Line 176 in 52603a7

return &transportProto, &securityProto, &probeProto, fmt.Errorf("only TCP and UDP are supported for Azure LoadBalancers")

CloudStack: it accepts only "TCP" or "UDP". Otherwise it returns with error.

kubernetes/pkg/cloudprovider/providers/cloudstack/cloudstack_loadbalancer.go

Line 455 in 52603a7

return nil, fmt.Errorf("unsupported load balancer protocol: %v", port.Protocol)

GCE: it allows only "TCP" or "UDP" for external LB. Otherwise it returns with error. For internal LB there is no explicit protocol check.

kubernetes/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

Line 750 in 52603a7

if ports[0].Protocol != v1.ProtocolTCP && ports[0].Protocol != v1.ProtocolUDP {

OpenStack: my understanding is, that OpenStack supports SCTP, so I enabled SCTP in the OpenStack cloudprovider code with this PR
oVirt, Photon, vSphere: these require the registration of a "cloud provider" and it is then up to the "LoadBalancer" implementation of the cloud provider how it handles the different protocols.

pkg/controller/service/service_controller.go also have a nice warning at the point where it calls the EnsureLoadBalancer() function of the relevant cloud provider:
" // - Not all cloud providers support all protocols and the next step is expected to return
// an error for unsupported protocols"

kubernetes/pkg/controller/service/service_controller.go

Line 372 in 52603a7

// - Not all cloud providers support all protocols and the next step is expected to return

As I can see all the supported cloud providers follows this principle, and they handle unsupported protocol values in their code.

Of course we are happy to receive comments from Network SIG. Please help in inviting the relevant people here if the "sig/network" label is not enough for a heads-up.

[MaximProshin]

LKSCTP is not loaded by default in Linux OS that allows developers to run user-land SCTP in Linux. With your change LKSCTP will automatically be loaded and it will intercept all incoming SCTP packets which will lead to mess and basically aborting of SCTP associations. In other words, this change will not allow developers to run user-land SCTP in K8s cluster.

[janosi]

@MaximProshin Is there the same effect when those are in different network namespaces? I.e. when the user space SCTP implementation is used inside a pod that does not use the host network namespace?

UPDATE: If the network interface of the pod is not a tun/tap-like one, and the user space SCTP stack uses raw socket with IPPROTO_SCTP, then namespace will not save us. On the other hand, considering the world without containers, one had to ensure that such user space SCTP apps are not deployed on the same nodes with such applications that would trigger the loading of lksctp. It drives me into the direction of thinking about a way how nodes could be reserved for such user-land SCTP applications. Assuming of course, that those user-land SCTP applications are deployed as containers/pods that share the same kernel with other pods - i.e. not as e.g. Kata Containers or similar.

[dcbw]

I think you need to update the BUILD file in this directory to account for this import addition. run 'hack/update-bazel.sh' and I think it'll do that for you.

[dcbw]

Same here, bazel update.

[dcbw]

mis-indented whitespace; also there should be a single space after the // in comments.

[janosi]

@dcbw Thank you for the comments! Updated with hack/update-gofmt.sh and hack/update-bazel.sh. Also did a rebase to the current master.

[MaximProshin]

Yes, namespaces don't help to isolate user-land SCTP from LKSCTP so there should be a way to not load/unload LKSCTP at least. If I compare K8s with native Linux OS, I would say the PR is non-backward compatible as previously LKSCTP wasn't loaded by default while with K8s it will be so.

[janosi]

@MaximProshin My understanding is that lksctp is not loaded as long as no SCTP Service (with Cluster IP or NodePort) is created, because of the on-demand loading. Once someone creates an SCTP Service the module is loaded, but it is an explicit request from the user. Just like when a user started a native Linux process that wanted to use SCTP.
If my understanding is wrong, please point to the place that triggers lksctp loading right at node startup, i.e. before there is an explicit request for that. Thank you!

UPDATE: we tested the module load case explicitly with the current code proposed here. My understanding is right: the current solution does not trigger the loading of the SCTP module by default(e.g. when kube-x is loaded at bootup). The SCTP module was loaded only when an SCTP Service was created. So, from this perspective if someone upgrades her k8s to a version that includes this feature, she will not experience any problems as long as she does not create SCTP Services explicitly.
This was about backward compatibility.
Starting from this stand we can discuss what we can do in order to have both SCTP Servies with ClusterIP/NodePort and userspace SCTP stack supported at the same time on the same cluster.
I think, we can agree, that these were not supported earlier either - i.e. if I had an application that used userspace SCTP and then I started another app that used kernel SCTP on the same node, then the result was the same problem. I.e. the solution, most probably, was to dedicate nodes to either of these SCTP users, i.e. to achieve that they are not deployed on the same nodes. Here we have the same challenge: to achieve that on the nodes where userspace SCTP apps are to be deployed we should not use the kernel level SCTP on any way.

[janosi]

@thockin I created a KEP, the PR is at kubernetes/community#2276

[thockin]

Thanks for the KEP. This also needs discussion on sig-network. A lot of significant issues in there.

[janosi]

Sure, we can take it to the next SIG Network meeting. Am I allowed to write it on the agenda?

[janosi]

Hopefully this PR kubernetes/test-infra#9161 solved the GKE test issues on 27th August 2018, so
/retest

UPDATE (30 minutes later): that PR is not in effect yet. The retest failed.

[thockin]

/lgtm
/approve
/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: janosi, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• api/OWNERS [thockin]
• cluster/gce/gci/OWNERS [thockin]
• cmd/OWNERS [thockin]
• docs/api-reference/OWNERS [thockin]
• pkg/apis/OWNERS [thockin]
• pkg/cloudprovider/providers/azure/OWNERS [thockin]
• pkg/cloudprovider/providers/gce/OWNERS [thockin]
• pkg/controller/endpoint/OWNERS [thockin]
• pkg/controller/service/OWNERS [thockin]
• pkg/features/OWNERS [thockin]
• pkg/kubectl/OWNERS [thockin]
• pkg/kubelet/OWNERS [thockin]
• pkg/master/OWNERS [thockin]
• pkg/printers/OWNERS [thockin]
• pkg/proxy/OWNERS [thockin]
• pkg/util/ipset/OWNERS [thockin]
• pkg/util/ipvs/OWNERS [thockin]
• staging/src/k8s.io/api/OWNERS [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[janosi]

/retest

[janosi]

/retest

[janosi]

/retest

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions here.

[janosi]

@thockin Thank you for guiding me through this adventure, for the review, and for the approval!
@MaximProshin @danwinship @dcbw @KomorkinMikhail @matrohon @m1093782566 Thank you for your comments and reviews!

[justaugustus]

Linking the features issue for this: kubernetes/enhancements#614

[yujuhong]

I don't think this gets enough attention from the sig-node. This adds a new SCTP protocol to CRI and may require changes in the CRI shims.

@kubernetes/sig-node-api-reviews @kubernetes/sig-node-pr-reviews @mrunalp @Random-Liu @resouer @feiskyer

[feiskyer]

On the Kubernetes side we solve this problem so, that we do not start listening on the SCTP ports defined for Servies with ClusterIP or NodePort or externalIP, neither in the case when containers with SCTP HostPort are defined. On this way we avoid the loading of the kernel module due to Kubernetes actions.

@yujuhong Should we skip SCTP port mapping in kuberuntime instead?

[janosi]

@feiskyer @yujuhong I wonder whether the same logic could be applied on the CRI side which was selected for the load balancers: k8s itself does not restrict the usage of SCTP as protocol for load balancer reuqests, and it is the task of the cloud provider's load balancer plugin to filter unsupported protocols.
For example, AFAIK, Docker supports SCTP for host port binding.
Also, I wonder if a clear and straightforward rejection of a request with an unsupported protocol would be better from the user's perspective than just skipping that.