Upgrading a clean cluster 1.27 to 1.28 - one of the nodes stuck in emergency mode #1362
Unanswered
asafbennatan
asked this question in
Q&A
Replies: 2 comments 3 replies
-
@asafbennatan You are in HA, so just turn off and turn back on the node. Normally it should work. |
Beta Was this translation helpful? Give feedback.
1 reply
-
@mysticaltech i just discovered this is a more serious issue as this happens when nodes are automatically upgraded , |
Beta Was this translation helpful? Give feedback.
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Description
i have installed the cluster on 1.27 after it was done without doing anything else i have upgraded it to 1.28 (~3:42 UTC)
all nodes were up and running except one ( see screenshot).
i used hetzner console to have a look at that node , terminal is stuck in emergency mode (see screenshot 2)
i pressed 'Enter' and everything started and the node is now online again and upgraded (this was at 6:41 UTC).
upon following the recommendation by the OS looking into
journalctl -xb
(output_reducted.txt attached) i see that the root cause of the issue as far as i gather is that/boot/writable
could not be mountedany idea why this would happen?
Kube.tf file
Screenshots
status after upgrade:
![image](https://private-user-images.githubusercontent.com/8994764/332706857-1558f29c-6014-46d6-b752-da3164b4aa54.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2MDY4MzcsIm5iZiI6MTcyMTYwNjUzNywicGF0aCI6Ii84OTk0NzY0LzMzMjcwNjg1Ny0xNTU4ZjI5Yy02MDE0LTQ2ZDYtYjc1Mi1kYTMxNjRiNGFhNTQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcyMiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MjJUMDAwMjE3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9M2M0MmY3MzdlMjIyZWM2YmZlY2UxZmExODMwOWIwYjA3ZDUxZjJhN2E2NjkxNjJkMTdjZWYyMjVhNGZlNjY0ZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.J4uqObpR6l1VK_lDxAD_p-jo9k8qixesDK8BlWTM8XA)
stuck at emergency:
output_reducted.txt
Platform
Linux
Beta Was this translation helpful? Give feedback.
All reactions