Health check does not seem to work on shard-worker when redis is down #1627
Comments
|
Buildfarm is currently intolerant to redis configuration changes. Redis clusters that change IPs/names are not reflected in server/worker grouped communication. Its something I've been meaning to experiment with tolerating, but the current guidance is to only run buildfarm on stable redis clusters, and to bring it down when clusters' layouts (including slot allocations) change. A number of modifications will need to be made to have the entire cluster respond correctly to reconfigured clusters. |
|
Thanks for explanation @werkt, from what I've observed is that if I restart buildfarm after redis comes back up, it starts working as usual. I can give a demo if your up for zoom call. One more anecdotal data point, we have old internal buildfarm build (before 2.8.0) which closes the tcp port if redis goes down. This allows me to use tcp liveness probe. In case of redis restart buildfarm gets restarted by kubernetes. Edit:
For redis we use kubernetes ClusterIP service which is like load balance with static IP address. We also pass redis address as domain (f.e.: |
|
I think I see - you're questioning the discrepancy between 'No available workers' and the threefold service availability on the workers. The problem needs to be looked at on the servers first - conditions in the cluster are leading to them, not the workers, responding with UNAVAILABLE. The servers need to see the manifest of current storage workers in redis to read/write blobs. I don't know if a failing connection to redis will inspire it into something other than the UNAVAILABLE behavior from above. |
aisbaa
changed the title
|
I think we are going away from the main problem I was referring to. Which is buildfarm shard-worker is not healthy (as observed in the dashboard), but liveness and readiness probes report that shard-worker is fine. Manually sending curl request to worker-shard readiness/liveness endpoit returns 200: |
|
200 on the prometheus endpoint doesn't necessarily indicate 'fine', but I agree that there's no presentation difference on any public interface that a worker can't talk to the backplane if it happens to be down. Ostensibly, the worker is fine, and coping with the fact that it can't talk to redis at a particular moment. Starting a worker does fail currently if it cannot connect to redis, which happens because backplane initialization was never made tolerant to it. With little effort, I could actually make it start cleanly and function normally, without any connection available to redis. So the statement "Health check does not seem to work..." is not accurate: the worker is healthy. Redis, hosting the backplane, is not. What do you want the worker to do differently when the backplane cannot be reached? |
|
Thank you for explanation, it does make sense. I initially though that redis is required for all operations and killing it makes the cluster unavailable. To add to that, I did find one worker in bad state and prometheus endpoint was not responding for that worker. So the health checks must be working too. Do you know a simple way to put a worker in a bad state so that it would report that it is not healthy? |
|
I've got a new data point. We are using automated GKE node pool upgrade, which happened over the weekend, which also restarted redis. While servers have to survive the redis restart, workers failed to rejoin servers.
Health check says that everything is fine: Logs indicate that worker failed to reconnect to redis: Bazel build completes, but gives us these warnings: Health checks are configured as follows: |
I would love that worker would report that it is not healthy if it can not talk to backplane, in that case kubernetes would restart it.
Is there a flag to tell worker that it must connect to redis and it must exit if it fails to do so? |
Without the below - where we shut down in the case where our connectivity to redis can't be established (any node), then we need to have a signal light that you'd get to see - I'd expect this to be on the prometheus endpoint, but will require more than just querying the endpoint and checking for 200: the specific metric we select to indicate connectivity failures would need to be observed.
I'll interpret this as 'can we make one', because there isn't one currently. Answer is yes. |
|
A prometheus metric would work, I could add a shell script that greps for metric and checks the value. As an alternative, would you consider GRCP health check to provide this signal. I think it would work nicely with potential flag/setting to report not healthy if connection to redis is lost. |
|
@aisbaa I think you are facing this problem #1552 Workers attempt to register to Redis at 10-second intervals. If Redis goes down, the connection request fails, and since the exception is unhandled, the thread responsible for registering the worker terminates. Applying the above pull request should resolve your issue.
In this situation, the worker can still communicate with the backplane; it's just that the thread responsible for registering the worker to the backplane has terminated. |
I've noticed that when redis gets restarted workers become unavailable for 5-10 minutes. Despite workers not working properly, they still reports that its being healthy. I tried 3 approaches for health checks: is tcp port is open, is http request on /metrics, grpc health check request, all of them indicate that service is healthy.
While trying to build a target with bazel it reports that no workers are available:
After restarting workers build completed with no warnings.
The text was updated successfully, but these errors were encountered: