1.24.0 loadbalancer health probe protocol issue when using ingress-nginx

[sevenam]

What happened:
After upgrading cluster to 1.24.0 external access through the loadbalancer stopped working.
Turns out the reason was because of the health probes of the load balancer.
These were set to use the protocols HTTP and HTTPS instead of TCP all of a sudden.

How to reproduce it (as minimally and precisely as possible):

• Create new cluster at version 1.24.0 or 1.24.3.
• Do helm install quickstart ingress-nginx/ingress-nginx
• Try the external IP and verify nginx cannot be reached
• Change the load balancer health probe protocol to TCP
• Try the external IP and verify nginx can be reached

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.24.0
• Size of cluster (how many worker nodes are in the cluster?): 1
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.): ingress-nginx
• Others:

[ghost]

Hi sevenam, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[wangyira]

@phealy could you take a look at this?

[emilverwoerd]

Could someon give an update on this issue since we are having the same issue and we are now forced to manually configure the loadbalancers to use TCP probing.

We deploy the nginx ingress controller through a helm chart and would like to know if there is an option to change it from there in the configuration. I can't find an additional service annotation which allows to change to TCP

[sajidhawk]

The new version of nginx ingress controller added appProtocol and its probe path has to be /healthz;
The new version of cloud-controller-manager added HTTP probing with default path / for appProtocol=http services.

Suggested solution and steps to mitigate your issue:
1. Take a Backup of the yaml of your service
kubectl get svc ingress-nginx-controller -n ingress-basic -o yaml >> backup_service_nginx.yaml
2. Edit your Nginx Service to modify the annotation
o kubectl edit service -n
o Add the following in the notations section: service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

This should resolve your issue.

[sevenam]

Thank you @sajidhawk - just tried this and can confirm it works for us. The health probe path for the load balancer was changed from / to /healthz

[emilverwoerd]

For us it also seams to work changing the path from / to healthz