upgrade k3s from 1.24.x to 1.25.x #7846
Labels
Comments
|
What is |
|
@brandond this the endpoint for kube-api - it's network loadbalancer from AWS. The problem is - that I see that it logs, but node already in cluster and connected. And that error is gone - once old nodes(1.24.x) removed from cluster. |
|
Just because the node is already in the cluster, doesn't mean that it skips validating the cluster connection on startup. Are you using health-checks on that load-balancer? I suspect that it's trying to send the request to a node that is currently down for maintenance or otherwise unavailable. The fact that the error goes away once the old nodes have been removed from the LB would seem to confirm this. |
|
@brandond it's not related definitely to the load balancer. I created new cluster - removed node from loadbalancer, but leave it inside k8s cluster. logs on old node 1.24.8: log from new node 1.25.11: node list list from kubectl also secret k3s-serving contains old node in a list (which is 10.8.16.118): I tried upgrade etcd nodes first and than masters - nothing changed, |
Yes, there is not currently any logic to prune CNs from the certificate
It looks like you're trying to run kubectl on a node that has been removed from the cluster? If so, this is expected behavior. If not, I'd be curious to see logs from the remaining etcd nodes. As long as you still have quorum, etcd leadership should not matter - leadership will be transferred to another node within milliseconds. |
|
@brandond no, I'm trying to run from any master node, status of etcd looks fine from etcdctl |
What you're reporting doesn't make any sense. The only reason that you should ever get Can you attach the complete k3s service logs from the etcd nodes in question? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environmental Info:
K3s Version:
1.24.8(but also tested on other)
Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04.2 LTS
AMD EPYC 7000
CPU: 2
Memory: 8
AWS m5a.large
Cluster Configuration:
3 etcd nodes
2-3 master nodes(usually 3, but tested 2 also)
Describe the bug:
During upgrade from 1.24.8 to 1.25.11 cluster is not stable and in degraded state.
Doing cordon and drain one by one master nodes to upgrade to newer version.
During last node replacement
Until old revision still exists in cluster
Steps To Reproduce:
explained on the above
Expected behavior:
Successful replacement of the nodes.
Additional context / logs:
On the middle of the replacement logs almost same as described here
#7123
The text was updated successfully, but these errors were encountered: