-
-
Notifications
You must be signed in to change notification settings - Fork 336
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Autoupgrade nodes seems to lead to not ready nodes that need manual reboots #1298
Comments
@sharkymcdongles No idea what could be happening, but I would suggest using our default cilium config instead. So remove cilium_values and try again. |
This is a recurring issue I noticed over the last couple of weeks, still investigating. It's most likely something related to all of our custom networking + microos + hetzner. For the time being, disable autoupgrades. |
And getting cilium to work well on Hetzner is super tricky, hence my above suggestion. |
I took what is documented here: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/README.md#examples Sadly, given the hetzner IP blacklist bs, using an egressgateway is the only way to ensure the cluster works with autoscaling because many images are stored in ghcr and hetzner ips are randomly blocked there. It's also needed for things like SMTP since many SMTP providers also block hetzner IPs. I will do some digging and see if maybe I can find out why this happens. I disabled autoupgrades now for both nodes and k3s yet some nodes still had the same behavior. So I did some digging and it looks like it still tried to do an upgrade leading to the NotReady nodes situation again. When I change autoupgrades to off does it not reflect it for already provisioned nodes? Do I need to remove kured? Checking the logs I see the upgrade runs then the CPU gets locked and NetworkManager gets stuck fully killing the networking for the node since it never recovers. I then see all these CPU stuck errors.
I am attaching my full log file from when this happened to see if maybe someone here can shine some light on it. I wonder if maybe the networking can't handle autoupgrades or updates given my settings? No idea tbh, but it'd be nice if this were solved or someone knew. I will continue researching on my end, but I think more heads are better than 1. |
It's possible. So try switching to default networking settings, remove |
@sharkymcdongles Any updates, did the suggestion work? |
@mysticaltech I haven't moved back to default cilium settings yet because I am working on a new way of handling images not pulling without the egress gateway. I am currently evaluating using a squid proxy as a replacement. Since disabling autoupgrades though I have had no further issues. |
Ok, great! The proxy solution sounds awesome. Don't hesitate to share in due time if you see fit. We are narrowing down the automated upgrade issues in other threads, so will close this one for now. |
Description
I notice many nodes become NotReady and don't come back without manually rebooting them from the CLI or cloud console.
Kube.tf file
Screenshots
No response
Platform
Linux
The text was updated successfully, but these errors were encountered: