New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow to disable loadbalancer health probes #1394
Comments
Can you provide a use case/business justification for completely disabling all health probes? |
Sure. Our use case is that we're exposing jenkins jnlp. The health probes flood on the one hand the log files with unnecessary log entries and the other more important factor seems to be that the health probes are "too aggressive" and disconnect the connection where it should still be ok and jenkins is without that able to keep the connection open. Right now we did the workaround from here: https://stackoverflow.com/a/54257960 : Basically changing externalTrafficPolicy to Local and adding an explicit healthCheckNodePort. Since we did that change the connection stays very stable where before it got interrupted every couple hours. |
I also would like to be able to disable all health probes. In our case it is log flooding (and our developers hate it analysing logs in case of issues ....). But also just to have an option. Not really sure what is the a use case/business justification to enforce health probes ? |
We also ran into similar issue. We were deploying an application that was listening for TCP connections on a specific port and then triggered an event when a connection was made. The health probes were triggering our events and as a result were spamming our logs with fake errors. |
For us this annotation would be helpful as well. We forward the TCP traffic to an outgoing connection. This is charged by bandwidth. The Healthprobes cause significant costs here. Therefore we had to use the workaround mentioned by @tarioch |
Any progress on this ? |
Action required from @Azure/aks-pm |
Action required from @Azure/aks-pm |
1 similar comment
Action required from @Azure/aks-pm |
another use case is using bitnami helm chart for mysql. Health probes flood the log with |
another use case is using the load balancer for udp services with no http or tcp endpoint. |
Almost the same case here. I have a raw socket that I don't want spammed. |
Any update on this? |
We are experiencing this issue as well. We have an mqtt port which is behind a loadbalancer. This mqtt port requires authentication and the health probe doesn't provide the authentication (of course) nor the correct protocol which results in logging (in the business application) of a faulty incoming request. |
I have a similar requirement. I'm hosting an |
This is currently a severe blocker for our deployment. We have a service exposing non-traditional protocols like websockets and custom communication protocol over TCP. The health probe is sending some data instead of empty netcat, so every few seconds there is a exception and stack trace in our logs. I understand that disabling health probe for ports is not a best practice, but it's fast solution to our issue discussed here. Other solution would be to allow us specify custom probe just as Kubernetes allows via readinessProbe and livenessProbe configuration. I propose simple LB annotation Example
I dug up some other annotations realted to health probes here, but that doesn't seem to work or I don't understand, what it does.
|
Would love to have a way to disable load balancer health checks for specific ports, use case for example is GRPC Ports don't like when TCP things probe them and do not ask for the GRPC Preface. This causes a ton of Log flooding which is just noise. Another valid option would be instead to allow the specific configuration to have a different healthcheck like a http healthcheck to the downstream service instead of checking the GRPC TCP port for availability. |
Hi all |
Putting LoadBalancer in front of a HTTP server as many have done above you need to be aware of following. A better solution is to consider a Ingress controller when dealing with HTTP. |
For my infrastructure it requires multiple rules across different port, so if I need to have multiple copies of my microservices, I need multiple copies of ingress-controller for TCP forwarding. As a result, I turned into the Azure CNI provided After that somehow the client was experiencing intermittent |
I had the same problem and was able to disable the health probe for my sftp server port with this annotation: From the docs: https://cloud-provider-azure.sigs.k8s.io/topics/loadbalancer/#loadbalancer-annotations
Where I think this issue can be closed as disabling health probes are already supported. |
This was always the possibility or it was recently added as new function to AKS LB? |
The doc states that it's possible since AKS Version v1.24. I don't know when the v1.24 Version was released. |
Currently there is no way to disable the loadbalancer health probes. It would be good if an annotation could be add to allow to disable the health probes. Either globally or for some of the ports.
The text was updated successfully, but these errors were encountered: