Azure Kubernetes publick8s suffers from SNAT port exhaustion: network slowness

[dduportal]

Service(s)

Azure

Summary

The AKS cluster publick8ssuffers from SNAT port exhaustion since around 1 month (example below for the last 24 hours):

It causes the following problems:

• Pagerduty is spamming us with false positive alerts reporting unusual response time from repo.jenkins-ci.org and reports.jenkins.io services while other probes (on other clusters) are reporting valid response times.
• The rest of the public network (including ci.jenkins.io and its agents, even on the peered network on the sponsored subscription,) is slowed down:
  • ci.jenkins.io jobs on Windows agents are much slower than 21 days ago #3904 (even if the network slowness is not the only reason)
  • Mirrorbits got pods restart on publick8s #3799 (mirrorbit pods are failing when trying to scan mirrors)

Reproduction steps

No response

[dduportal]

• This problem rings a bell in my memory: https://github.com/jenkins-infra/documentation/blob/5971469e62d93184ad4ffa34b969f53a1c7af45c/archives/aks-1.19.md?plain=1#L101 => we did had and solved it on the former prodpublick8s (replaced by publick8s since then).
  • We could check the method we used (allocating a public IP pool prefix to the loadBlancer used for outbound connectivity)
  • But this time let's track it and even better: as publick8s is terraformed, let's see if we can use config from https://www.danielstechblog.io/detecting-snat-port-exhaustion-on-azure-kubernetes-service/
  • AKS clusters now support NAT gateway outbound method but we cannot migrate an existing AKS cluster to it: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections
  • WiP: reading https://www.danielstechblog.io/preventing-snat-port-exhaustion-on-azure-kubernetes-service-with-virtual-network-nat/ to understand better the concept of Virtual Network NAT which is distinct from a "User NAT Gateway"

[dduportal]

• The ci.jenkins.io networks could be moved out of the publick8s network to separate outbounds routes:
  • Since agents are using the new subscription, we could set it up) to use a NAT gateway for outbound like we already did with trusted.ci and cert.ci
  • ci.jenkins.io is currently using public Ip outbound: moving its controller VM/disk to the secondary subscription would ensure:
    • Complete removal of the public vnet peering (and separate concerns) as ci.j would be on another network
    • More usage on the subscription billing (~450$ monthly)

[dduportal]

Update:

• Loadbalancer tuned for both publick8s and privatek8s: feat(publick8s,privatek8s) set up outbound LB to support more SNAT connections azure#579
  • It required hotfixes hotfix(publick8s) correct outbound LB SNAT port azure#580 hotfix(privatek8s) correct outbound LB SNAT port azure#581 and hotfix(publick8s,privatek8s) set up LB ports amount to valid values azure#582) as the specified max outbound port number must be:
    • Per VM
    • A multiple of 8 (or said differently, a division of the 64000 max ports available per public IP)

Next step:

• NAT gateway for ci.jenkins.io sponsorship network
• migration of ci.jenkins.io VM to the sponsorship network

[dduportal]

Update:

• ci.jenkins.io is now using a NAT gateway for its outbound traffic: if there are any remaning slowness it won't be related to SNAT port exhaustion

[dduportal]

Update:

* Loadbalancer tuned for both `publick8s` and `privatek8s`: [feat(publick8s,privatek8s) set up outbound LB to support more SNAT connections azure#579](https://github.com/jenkins-infra/azure/pull/579)
  
  * It required hotfixes [hotfix(publick8s) correct outbound LB SNAT port azure#580](https://github.com/jenkins-infra/azure/pull/580) [ hotfix(privatek8s) correct outbound LB SNAT port azure#581](https://github.com/jenkins-infra/azure/pull/581) and [hotfix(publick8s,privatek8s) set up LB ports amount to valid values azure#582](https://github.com/jenkins-infra/azure/pull/582)) as the specified max outbound port number must be:
    
    * Per VM
    * A multiple of 8 (or said differently, a division of the 64000 max ports available per public IP)

No SNAT port exhaustion detected in the past 12 hours as per Azure metrics:

[dduportal]

• The ci.jenkins.io networks could be moved out of the publick8s network to separate outbounds routes:

  * Since agents are using the new [subscription](https://github.com/jenkins-infra/helpdesk/issues/3818), we could set it up) to use a NAT gateway for outbound like we [already did with trusted.ci and cert.ci](https://github.com/jenkins-infra/azure/pull/567)
  * ci.jenkins.io is currently using public Ip outbound: moving its controller VM/disk to the secondary subscription would ensure:
    
    * Complete removal of the public vnet peering (and separate concerns) as ci.j would be on another network
    * More usage on the subscription billing (~450$ monthly)
  

Tracking the ci.jenkins.io migration in #3913

[dduportal]

While working on #3837 (comment), we saw the problem to re-appear due to additional nodes.

It sounds like we should add more public IP to increase the threshold. If it does not suffice, we'll have to plan a cluster re-creation during the Kubernetes 1.27 upgrade with a new subnet (and associated NAT gateway).

[dduportal]

Update:

• Opened a PR to increase the number of Public IPs: fix(publick8s) add more public IPs for outbound connection to avoid SNAT exhaustion azure#587
  • Required an hotfix as increasing the amount of IPv4 also required to specify IPv6 (set to 2): jenkins-infra/azure@f3ba3d8
• Once deployed, we'll have to retrieve the value of these public IPs and add it in https://github.com/jenkins-infra/kubernetes-management/blob/main/config/ldap.yaml

[dduportal]

Update:

* Opened a PR to increase the number of Public IPs: [fix(publick8s) add more public IPs for outbound connection to avoid SNAT exhaustion azure#587](https://github.com/jenkins-infra/azure/pull/587)
  
  * Required an hotfix as increasing the amount of IPv4 also required to specify IPv6 (set to 2): [jenkins-infra/azure@f3ba3d8](https://github.com/jenkins-infra/azure/commit/f3ba3d81e758a3a1d9d033d51ab99893d14a6d7a)

* Once deployed, we'll have to retrieve the value of these public IPs and add it in https://github.com/jenkins-infra/kubernetes-management/blob/main/config/ldap.yaml

• Current outbound IPs:

outbound publick8s:

- 2603:1030:403:3::106
- 2603:1030:403:3::217
- 20.85.71.108
- 20.22.30.9
- 20.22.30.74

outbound privatek8s:

- 20.22.6.81

• Updated LDAP: jenkins-infra/kubernetes-management@8a7f2bc

Let's check how is the week-end going

[dduportal]

Alas we still see SNAT port problem. Lets go the "add a NAT gateway but not explicitly" as described in https://www.danielstechblog.io/preventing-snat-port-exhaustion-on-azure-kubernetes-service-with-virtual-network-nat/ (and other internet posts)

[dduportal]

Update: opened jenkins-infra/azure-net#198, let's prepare this PR and check the SNAT metrics before and after deploying to confirm SNAT exhaustion disappear.

If it does, then we'll decrease the LB outbound setup IPs to pay less.

[dduportal]

Update:

• Dashboard for SNAT tracking created in Azure (Go to "Dashboard Hub" Section -> select the "ASK Outbound SNAT Connection" shared dashboard")

• Opened feat(gateways) add a NAT gateway to privatek8s for outbound requests azure-net#199 to try the migration on the privatek8s cluster before the public one

[dduportal]

Update: we'll delay the switch to a NAT gateway for after the 2.426.3 LTS release

[dduportal]

Update: we'll delay the switch to a NAT gateway for after the 2.426.3 LTS release

Let's go! Ref. jenkins-infra/azure-net#201

[dduportal]

Looks good for now:

[dduportal]

The fix was efficient: we see no more SNAT exhaustion 🥳

[dduportal]

Confirmed after the weekend: we can close: