You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.
We have a cluster that does autoscaling (EC2 instances that are stopped/started/restarted depending), and it appears that on occasion, a restarted machine will not be able to connect to peers, in part because they had hit their conn-limit with connections to machines that are no longer alive. I bumped up from the default conn-limit, but didn't know if running weave reset before shutting down a machine would fix this issue (and reclaim IP addresses used by that node).
It does not appear that running kubectl delete pod <weave-pod> runs weave reset (per slack discussion with @brb). I was wondering if adding a preStop hook (https://kubernetes.io/docs/user-guide/container-environment/#hook-details) to the weave daemon set to run weave reset would be sufficient here. I also know it would require adding /var/run/docker.sock as a hostPath mounted volume to the weave container in order to run weave reset.
The Allocating IP addresses section of the doc mentions that weave "also releases that address when the container exits." but I'm not sure that happens if it doesn't do a weave reset https://www.weave.works/docs/net/latest/ipam/
The text was updated successfully, but these errors were encountered:
Okay, sounds good thanks. I think it is safe to close this issue then.
I believe the multiple connections issue was from not waiting enough between restarts of the container. I haven't seen it reliably in our normal setup, just when I was doing some manual reboots and container restarts to investigate. The conn-limit and hosts connecting to old instances seemed to be the real issue.
Re: Multiple connections: For anyone who stumbles here, after doing more digging on the next node failure I believe that the root issue we were seeing was: #2617 as the weave-kube pod was coming up with exit status 1: iptables: No chain/target/match by that name. on v1.8.0 occasionally on node reboot. We bumped versions and don't believe the issue is recurring any longer (rebooted nodes a number of times in a row but will need time to be sure)
We have a cluster that does autoscaling (EC2 instances that are stopped/started/restarted depending), and it appears that on occasion, a restarted machine will not be able to connect to peers, in part because they had hit their
conn-limit
with connections to machines that are no longer alive. I bumped up from the default conn-limit, but didn't know if runningweave reset
before shutting down a machine would fix this issue (and reclaim IP addresses used by that node).It does not appear that running
kubectl delete pod <weave-pod>
runsweave reset
(per slack discussion with @brb). I was wondering if adding a preStop hook (https://kubernetes.io/docs/user-guide/container-environment/#hook-details) to the weave daemon set to runweave reset
would be sufficient here. I also know it would require adding/var/run/docker.sock
as a hostPath mounted volume to the weave container in order to runweave reset
.Partially related: #2797
The Allocating IP addresses section of the doc mentions that weave "also releases that address when the container exits." but I'm not sure that happens if it doesn't do a
weave reset
https://www.weave.works/docs/net/latest/ipam/
The text was updated successfully, but these errors were encountered: