Regression in winkernel proxier that causes stale load balancing proxy rules

[daschott]

What happened?

There is a regression in v1.24.0 (and above) that causes stale HNS load balancer proxy rules anytime a backend pod is deleted. Each subsequent deletion will leave behind an additional external VIP load balancing rule that references endpoints which no longer exist.

This can cause occasional connectivity issues and timeouts, if a stale load balancing rule is matched and it redirects traffic to an endpoint which no longer exists.

We are under the impression this may have been introduced here. (creds to @sbangari for identifying this.)

What did you expect to happen?

I expect all load balancing rules created by winkernel proxier (aka HNS load balancer) to be referencing valid backends (aka HNS endpoints)

How can we reproduce it (as minimally and precisely as possible)?

1. Create a Kubernetes cluster v1.24 or higher with Windows nodes.
2. Create a Kubernetes Service with type "LoadBalancer" on Windows.
3. Delete one or more pods of the Service.
4. Observe stale load balancing rule left behind referencing endpoint that was already deleted.

Anything else we need to know?

This issue can be discovered by establishing multiple connections to the service, after pods were deleted. Some of the requests will fail.

This issue can be monitored using the following script:
https://raw.githubusercontent.com/daschott/SDN/patch-1/Kubernetes/windows/debug/networkhealth.ps1

Execute as follows:
.\networkhealth.ps1 -OutputMode Stdout

It will print something along the lines of:
10/3/2022 8:18:28 PM <my_node_name> @{Problem=Detected 1 stale VIPs <my_service_vip> }

Another way to inspect this manually is using the following PowerShell:

$refs = get-hnspolicylist | select @{Name="VIP"; Expression={$_.Policies.VIPs}}, References | ? VIP -Like "my_svc_vip" | Select References
get-hnsendpoint | select IPAddress, ID

In the $refs, you would see multiple entries (indicating there are duplicate HNS load balancer proxy rules with the same VIP) and that some of the references are invalid (ie not showing up in the get-hnsendpoint output)

Workaround

1. You can force all pods of a given service onto a single Node. This will still cause duplicated load balancer entries, but the connections should all succeed as the right rule should be matched.
2. Use K8s version <v1.24.0
3. Restart-service kubeproxy. This will remove all the rules and re-create them.

Kubernetes version

1.24.0 and above

Cloud provider

Azure Kubernetes Service, but likely impacting others.

OS version

Windows Server 2019, Windows Server 2022

Install tools

n.A.

Container runtime (CRI) and version (if applicable)

n.A.

Related plugins (CNI, CSI, ...) and versions (if applicable)

n.A.

[daschott]

/sig windows

[jsturtevant]

/triage accepted

[jsturtevant]

/sig network

[marosset]

/milestone v1.26
/priority critical-urgent

[marosset]

/reopen
Let's keep this open until backport PRs merge and the fix is available in v1.24 and v1.25 builds.

[k8s-ci-robot]

@marosset: Reopened this issue.

In response to this:

/reopen
Let's keep this open until backport PRs merge and the fix is available in v1.24 and v1.25 builds.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jsturtevant]

backports are merged and patches were released
/close

[k8s-ci-robot]

@jsturtevant: Closing this issue.

In response to this:

backports are merged and patches were released
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.