Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node/instance connection not working anymore after node memory overload #41

Open
aptroost opened this issue Dec 4, 2022 · 0 comments
Open

Comments

@aptroost
Copy link

aptroost commented Dec 4, 2022

Problem
When a node is memory overloaded, the state of that node will be defined as unknown. Meanwhile the instance is running fine when looking in the Hetzner dashboard. The node is not working in Kubernetes and all running pods are stuck in either terminated or restarting state.
A terraform plan would return no changes.

Tested solutions that do not work
I tried a drain and delete for the node and removed the Hetzner instance manually.
After that I retried terraform plan and apply, that would successfully recreate the Hetzner instance, but the instance was not joined to the kubernetes cluster (node not added to cluster).

To be researched

  • Potentially when a Kubernetes node is running on the instance and crashes, this provider will not restart the node on that instance. This can be tested by memory overloading the node while checking the instance.
  • When removing the node and instance manually, a kubectl join command seems not to be executed on the instance. This can be tested by manually executing this process while monitoring the processes on the new instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant