allow node selector filter for backends

[daniel-garcia]

Currently, creating a LoadBalancer service attaches every node that runs kubelet as a backend to created backend set.

In our cluster configuration we want the backend traffic to be restricted to a subnet of hosts. If the configuration allowed a tag specification, the matching nodes could be filtered by label.

[prydie]

Hi Daniel,

You should be able to achieve the desired result within Kubernetes rather than via extension of the CCM. I suggest labeling the Nodes that you want to restrict the traffic to, useing a corrosponding label selector on the Deployment that you are exposing via the load balancer, and setting externalTrafficPolicy: Local on the Service. Though all nodes in the cluster will be added to the BackendSet the health check will fail for nodes on which the Deployment is not running (i.e. the unlabeled nodes) and traffic won't be routed via the other Nodes in the cluster.

More information is available in the Kubernetes documentation here.

[prydie]

Hi @daniel-garcia, I'm going to close this issue. If the above soltion isn't satisfactory please do let us know and we'll re-open and explore further.

[daniel-garcia]

The given approach is not optimal. This approach would cause the status of the LB to be marked at degraded because possibly many backends would be failing. The backends could be restricted using a configuration such as a regex or a node selector expression. This would allow CCM to only add nodes that match the criteria instead of adding all possible clusters to the backend set.

[prydie]

@jhorwit2 @owainlewis thoughts? I’m still inclined to stick to the standard approach rather than adding an additional custom implementation.

[jhorwit2]

@daniel-garcia This isn't going to be possible to support as is and would require upstream changes.

Currently, backends in the cloud are updated on a set interval; if we made this change that would mean you run the risk of having pods on a node that don't immediately start receiving traffic. The default interval is every 100s if i'm not mistaken. It's also forking essentially the common interface we all follow for those core features/how they work.

If you feel strongly about this I'd suggest opening an upstream issue and CCing me on it. I'll ping the right people to start discussions. We'd want to change how the underlying code that all the CCM's share works.

The backends could be restricted using a configuration such as a regex or a node selector expression.

Regex isn't "standard k8s" so we wouldn't want to go with that. We could go with node selectors; however, that'd be restricted to annotation based values which mean only key=value and not more complex expressions that are possible.

[owainlewis]

@prydie @jhorwit2 solved this one. See https://kubernetes.io/docs/reference/feature-gates/ ServiceNodeExclusion: Enable the exclusion of nodes from load balancers created by a cloud provider. A node is eligible for exclusion if annotated with “alpha.service-controller.kubernetes.io/exclude-balancer” key.