Services with health checks sometimes get stuck in degraded/initializing

[cjellick]

In building my kubernetes stack, when I added health checks, about half the time, the services/containers never got passed the degraded/initializing phase.

[cusspvz]

This happens to me with hosts that are going to be or are in "Reconnecting" state.

[alena1108]

@cusspvz we've put a bunch of fixes related to healthchecks + hosts in reconnecting state combination - all will be available in next rancher release. The problem @cjellick is facing, is different. Today we don't support health checks for instances having Host networking. So if the health check is specified for such instance, the instance will be in initializing state forever.

[janeczku]

This just happened to me with Rancher DNS service on a new DigitalOcean host and i can't get it working. Can you advise how to troubleshoot/debug the stuck in initializing sate?
Update: This was actually caused by IPSec connectivity issue (at least in my case): #2901 (comment)

[will-chan]

not enough information here so closing issue.

[raphink]

I'm using rancher 0.56.1 and seeing this behavior:

Services I've set health checks for start fine, work fine (they're even behind a LB and working good). I see the health checks in the container logs passing (and returning a 200 code), but both services are marked as "Initializing".

[raphink]

Also, the load-balancer itself (using rancher/load-balancer-service, with a health-check) is marked as "Initializing" as well, even though it is working fine.

[raphink]

and the bottom of the load-balancer logs (one of the containers):

[raphink]

Edit: after a while, the mapserver service was not load-balanced anymore. Stop and restarting it put it behind the LB again. That container is an Apache container. Here's an excerpt of its logs, showing the health check pings passing:

[raphink]

I removed switched a TCP check instead of an HTTP check (removed the request_line: parameter) and I'm still seeing the same behavior (services marked as "Initializing")

[raphink]

Upgrading services to remove the health checks completely sets them to the "Active" state. The load-balancer however is still marked as "Initializing", even after a restart.

[raphink]

OK, that turned out to be a firewall issue. When I added a rule to my security group to allow all traffic on all protocols from all sources, the health checks passed. However, allowing them from within the security group of the nodes or from the rancher server did not need to suffice.

Which firewall rules are actually necessary for health checks to work properly?

[deniseschannon]

Health check ports for load balancers are port 42.