Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weave not working correctly leads to containers stuck in ContainerCreating #3384

Closed
Raffo opened this Issue Aug 23, 2018 · 55 comments

Comments

Projects
None yet
8 participants
@Raffo
Copy link

Raffo commented Aug 23, 2018

What you expected to happen?

Weave should not have memory of previous removed nodes as this can cause ax exhaustion of IPs.

What happened?

Some containers of the cluster were in status ContainerCreating and could never transion to a Running status. We could see by describing one of the pods that they were reporting Failed create pod sandbox. Here is a list of similar issues which could still be unrelated:

In our cluster we scale down the nodes every night to save money, by changing the size of the Autoscaling Group in AWS (it's a kops cluster).

We saw the following in weave containers:

for i in $(kubectl get pods -n kube-system | grep weave | awk '{ print $1}'); do kubectl get pods $i -o wide -n kube-system; kubectl exec -n kube-system $i -c weave -- /home/weave/weave --local status connections; done

<- 11.10.53.254:59238   established fastdp 9e:51:84:a9:2b:99(ip-10-11-53-254.eu-west-1.compute.internal) mtu=8912
<- 11.10.125.51:52928   established fastdp aa:52:36:e7:8d:d3(ip-10-11-125-51.eu-west-1.compute.internal) mtu=8912
<- 11.10.95.88:60391    established fastdp e2:a6:ae:06:8f:d1(ip-10-11-95-88.eu-west-1.compute.internal) mtu=8912
-> 11.10.51.247:6783    failed      cannot connect to ourself, retry: never
<- 11.10.53.254:33762   established fastdp 9e:51:84:a9:2b:99(ip-10-11-53-254.eu-west-1.compute.internal) mtu=8912
-> 11.10.51.247:6783    established fastdp 6a:a4:ca:68:f4:02(ip-10-11-51-247.eu-west-1.compute.internal) mtu=8912
<- 11.10.95.88:58856    established fastdp e2:a6:ae:06:8f:d1(ip-10-11-95-88.eu-west-1.compute.internal) mtu=8912
-> 11.10.125.51:6783    failed      cannot connect to ourself, retry: never
-> 11.10.51.247:6783    established fastdp 6a:a4:ca:68:f4:02(ip-10-11-51-247.eu-west-1.compute.internal) mtu=8912
-> 11.10.125.51:6783    established fastdp aa:52:36:e7:8d:d3(ip-10-11-125-51.eu-west-1.compute.internal) mtu=8912
-> 11.10.95.88:6783     established fastdp e2:a6:ae:06:8f:d1(ip-10-11-95-88.eu-west-1.compute.internal) mtu=8912
-> 11.10.53.254:6783    failed      cannot connect to ourself, retry: never
-> 11.10.125.51:6783    established fastdp aa:52:36:e7:8d:d3(ip-10-11-125-51.eu-west-1.compute.internal) mtu=8912
-> 11.10.51.247:6783    established fastdp 6a:a4:ca:68:f4:02(ip-10-11-51-247.eu-west-1.compute.internal) mtu=8912
<- 11.10.53.254:55665   established fastdp 9e:51:84:a9:2b:99(ip-10-11-53-254.eu-west-1.compute.internal) mtu=8912
-> 11.10.95.88:6783     failed      cannot connect to ourself, retry: never

This is NOT significant. It's fine that the nodes say that they can't connect to ourself or at least we see this error in the status connection command of the weave CLI even on a working cluster.

What is more interesting is the output of status ipam:

kubectl exec -n kube-system weave-net-dcrj2 -c weave -- /home/weave/weave --local status ipam
9e:51:84:a9:2b:99(ip-10-11-53-254.eu-west-1.compute.internal)        7 IPs (00.0% of total) (7 active)
ba:3e:73:6a:13:c7()                        256 IPs (00.0% of total) - unreachable!
a2:6a:83:e2:d2:7e()                         32 IPs (00.0% of total) - unreachable!
32:a6:83:f6:c0:25()                       1024 IPs (00.0% of total) - unreachable!
8e:19:b3:42:4a:ec()                       2048 IPs (00.1% of total) - unreachable!
ba:bb:8e:64:d8:07()                       4096 IPs (00.2% of total) - unreachable!
da:f0:0a:b5:31:58()                     524288 IPs (25.0% of total) - unreachable!
ae:fc:8e:74:74:53()                       2048 IPs (00.1% of total) - unreachable!
3e:a3:6c:2c:68:9c()                         16 IPs (00.0% of total) - unreachable!
3e:bc:b5:42:15:66()                         32 IPs (00.0% of total) - unreachable!
26:87:a6:1f:4c:82()                       8192 IPs (00.4% of total) - unreachable!
82:cf:4e:23:3f:73()                       4096 IPs (00.2% of total) - unreachable!
ba:82:f4:d0:10:c5()                      32768 IPs (01.6% of total) - unreachable!
62:06:bf:fa:c8:b2()                       4096 IPs (00.2% of total) - unreachable!
5e:fd:cf:58:ce:01()                        256 IPs (00.0% of total) - unreachable!
5a:f7:3b:61:39:61()                         32 IPs (00.0% of total) - unreachable!
36:b5:90:80:65:88()                        512 IPs (00.0% of total) - unreachable!
36:91:10:1e:29:de()                       1024 IPs (00.0% of total) - unreachable!
62:0b:d4:f8:e1:51()                       4096 IPs (00.2% of total) - unreachable!
9a:7c:fa:51:3b:a9()                        192 IPs (00.0% of total) - unreachable!
da:c7:bd:46:98:c7()                       1024 IPs (00.0% of total) - unreachable!
e6:cf:6c:3e:fb:b0()                       2048 IPs (00.1% of total) - unreachable!
42:81:30:9e:df:0a()                        128 IPs (00.0% of total) - unreachable!
fe:77:8f:46:67:f4()                       1024 IPs (00.0% of total) - unreachable!
0e:85:43:e7:98:c2()                        512 IPs (00.0% of total) - unreachable!
3a:83:86:eb:df:da()                        128 IPs (00.0% of total) - unreachable!
16:41:a0:af:8c:3e()                        128 IPs (00.0% of total) - unreachable!
3e:8c:be:be:a7:0c()                         16 IPs (00.0% of total) - unreachable!
fa:88:5f:ea:c5:5f()                      65536 IPs (03.1% of total) - unreachable!
9a:ba:ce:4d:60:bd()                       1024 IPs (00.0% of total) - unreachable!
d6:ad:e3:03:aa:42()                         32 IPs (00.0% of total) - unreachable!
56:db:68:38:9b:5b()                         32 IPs (00.0% of total) - unreachable!
3a:0c:3c:e9:59:d8()                        128 IPs (00.0% of total) - unreachable!
b6:76:96:73:bc:6b()                       2048 IPs (00.1% of total) - unreachable!
1e:e8:8e:ad:fd:a9()                     262144 IPs (12.5% of total) - unreachable!
8a:f0:9a:e1:c7:29()                         32 IPs (00.0% of total) - unreachable!
e2:27:36:19:4e:c1()                      32768 IPs (01.6% of total) - unreachable!
0e:bf:ce:ac:ea:dd()                        256 IPs (00.0% of total) - unreachable!
8a:00:d6:3d:67:39()                        256 IPs (00.0% of total) - unreachable!
ae:03:57:54:c1:ec()                       2048 IPs (00.1% of total) - unreachable!
1a:0d:d2:ff:88:3b()                      32768 IPs (01.6% of total) - unreachable!
06:68:b6:87:48:75()                         64 IPs (00.0% of total) - unreachable!
9e:f4:4f:b3:77:07()                       8192 IPs (00.4% of total) - unreachable!
22:85:55:e9:07:e3()                         64 IPs (00.0% of total) - unreachable!
a6:cc:48:0b:42:8a()                        128 IPs (00.0% of total) - unreachable!
fa:2e:36:62:23:d9()                       1024 IPs (00.0% of total) - unreachable!
ae:c8:70:e0:23:22()                      49152 IPs (02.3% of total) - unreachable!
be:66:9a:85:fa:df()                         16 IPs (00.0% of total) - unreachable!
46:cb:ba:1c:b4:3a()                         16 IPs (00.0% of total) - unreachable!
fa:00:d3:e8:a4:f1()                      32768 IPs (01.6% of total) - unreachable!
8e:d7:cf:ff:97:69()                      16384 IPs (00.8% of total) - unreachable!
aa:52:36:e7:8d:d3(ip-10-11-125-51.eu-west-1.compute.internal)       19 IPs (00.0% of total)
fe:05:22:50:04:0a()                       2048 IPs (00.1% of total) - unreachable!
3e:91:da:4d:a9:ec()                     262144 IPs (12.5% of total) - unreachable!
82:a3:c7:f9:6d:e9()                        128 IPs (00.0% of total) - unreachable!
2e:8b:a6:cc:a7:19()                         32 IPs (00.0% of total) - unreachable!
2e:f7:59:91:b2:11()                       4096 IPs (00.2% of total) - unreachable!
c6:18:a6:97:97:4c()                      32768 IPs (01.6% of total) - unreachable!
56:ab:99:e9:91:fd()                      16384 IPs (00.8% of total) - unreachable!
7a:6d:41:17:b0:c3()                         20 IPs (00.0% of total) - unreachable!
c2:7f:f3:07:bf:48()                       2048 IPs (00.1% of total) - unreachable!
82:83:52:4f:34:f8()                     524288 IPs (25.0% of total) - unreachable!
6a:a4:ca:68:f4:02(ip-10-11-51-247.eu-west-1.compute.internal)        2 IPs (00.0% of total)
6a:09:6a:72:65:31()                         64 IPs (00.0% of total) - unreachable!
3a:fe:7d:61:b6:12()                         32 IPs (00.0% of total) - unreachable!
9e:93:78:0d:95:6f()                        512 IPs (00.0% of total) - unreachable!
a2:3e:3e:c8:40:34()                         16 IPs (00.0% of total) - unreachable!
82:68:49:b6:38:28()                       4096 IPs (00.2% of total) - unreachable!
c2:78:2d:27:b1:4d()                      16384 IPs (00.8% of total) - unreachable!
76:5f:e2:06:fa:35()                     131072 IPs (06.2% of total) - unreachable!

This seems to be telling us that most of the cluster is unreachable... which is making the CNI not work and containers can't start cause they can't get an IP address.
We verified that this was the case by reading the kubelet logs:

Aug 23 07:46:29 ip-10-11-53-254 kubelet[7383]: E0823 07:46:29.891765    7383 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Aug 23 07:46:29 ip-10-11-53-254 kubelet[7383]: E0823 07:46:29.891815    7383 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Aug 23 07:46:29 ip-10-11-53-254 kubelet[7383]: E0823 07:46:29.891832    7383 kuberuntime_manager.go:647] createPodSandbox for pod "nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Aug 23 07:46:29 ip-10-11-53-254 kubelet[7383]: E0823 07:46:29.891888    7383 pod_workers.go:186] Error syncing pod f6fe3f93-a6a6-11e8-80a5-0205d2a81076 ("nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)"), skipping: failed to "CreatePodSandbox" for "nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Aug 23 07:46:30 ip-10-11-53-254 kubelet[7383]: I0823 07:46:30.730025    7383 kuberuntime_manager.go:416] Sandbox for pod "nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)" has no IP address.  Need to start a new one
Aug 23 07:46:31 ip-10-11-53-254 kubelet[7383]: I0823 07:46:31.436352    7383 kubelet.go:1896] SyncLoop (PLEG): "nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)", event: &pleg.PodLifecycleEvent{ID:"f6fe3f93-a6a6-11e8-80a5-0205d2a81076", Type:"ContainerDied", Data:"da883b31b03187408bbee1b4642ba836932776977c200905fcb8e5f8cb9f4024"}
Aug 23 07:46:31 ip-10-11-53-254 kubelet[7383]: W0823 07:46:31.436438    7383 pod_container_deletor.go:77] Container "da883b31b03187408bbee1b4642ba836932776977c200905fcb8e5f8cb9f4024" not found in pod's containers
Aug 23 07:46:31 ip-10-11-53-254 kubelet[7383]: I0823 07:46:31.436465    7383 kubelet.go:1896] SyncLoop (PLEG): "nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)", event: &pleg.PodLifecycleEvent{ID:"f6fe3f93-a6a6-11e8-80a5-0205d2a81076", Type:"ContainerStarted", Data:"4deab2663ce209335c30401f003c0465401ef20604d32e2cfbd5ec6ab9b6b938"}
Aug 23 07:47:05 ip-10-11-53-254 kubelet[7383]: I0823 07:47:05.109777    7383 server.go:796] GET /stats/summary/: (3.458746ms) 200 [[Go-http-client/1.1] 11.10.125.51:38646]
Aug 23 07:48:05 ip-10-11-53-254 kubelet[7383]: I0823 07:48:05.027382    7383 server.go:796] GET /stats/summary/: (3.582405ms) 200 [[Go-http-client/1.1] 11.10.125.51:38646]
Aug 23 07:48:26 ip-10-11-53-254 kubelet[7383]: I0823 07:48:26.863628    7383 container_manager_linux.go:425] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service

In the logs above you can see has no IP address. Need to start a new one.

We believe that this is due to the fact that we continuously shut down the nodes of our cluster in the night by simply scaling the ASG to 0 and back to the original size in the morning. It looks like that kops/weave do not do any automatic cleanup, probably cause they don't have a chance.

From the weave documentation, it seems that we have to do something when the nodes exits, like mentioned in the official documentation. We still have to find a proper way to remove nodes from the Kubernetes cluster.

We did the reset by doing the following:

  • ssh into the EC2 instances (masters and workers) one by one and delete the file /var/lib/weave/weave-netdata.db. There is no need for a backup of that file
  • restart all the weave pods by deleting them, i.e.: for i in $(kubectl get pods -n kube-system | awk '{print $1}' | grep weave); do kubectl delete pod -n kube-system $i; done

This brought us back to a healthy state, that we could figure by running again the status ipam weave command:

k exec -it weave-net-47lhb -n kube-system -c weave /bin/sh
/home/weave # ./weave --local status ipam
9e:51:84:a9:2b:99(ip-172-20-53-254.eu-central-1.compute.internal)   524289 IPs (25.0% of total) (8 active)
6a:a4:ca:68:f4:02(ip-172-20-51-247.eu-central-1.compute.internal)   786411 IPs (37.5% of total)
aa:52:36:e7:8d:d3(ip-172-20-125-51.eu-central-1.compute.internal)   524307 IPs (25.0% of total)
e2:a6:ae:06:8f:d1(ip-172-20-95-88.eu-central-1.compute.internal)   262145 IPs (12.5% of total)

How to reproduce it?

Not sure, probably deleting lots of nodes from the cluster in a continuous way.

Anything else we need to know?

Versions:

$ weave version: 2.3.0
$ docker version

Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ uname -a
Linux ip-172-20-95-88 4.4.0-1054-aws #63-Ubuntu SMP Wed Mar 28 19:42:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Logs:

I don't have other logs to paste for the moment.

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Aug 23, 2018

The symptom looks very like #2797, which was fixed before 2.3.0.
We'd need the logs from a weave container starting after the point where it went wrong, to see why cleanup isn't working.
If there are no weave containers that start after the scale-down, then this is the same as #3372
(also noted at #3171).

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Aug 23, 2018

It's unfortunate that I lost the logs of the weave container, but it should be possible to replicate it (just startup and shutdown nodes, should be easy to script). I will give it a shot early next week.
Anyway, the weave containers were all started and Running.

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Aug 23, 2018

To be clear, it only runs cleanup on new pods starting, so if you know for sure you didn't start any new weave pods between scale-down and hitting the problem, then we don't need any more information.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Aug 23, 2018

Weave is deployed as a daemonset so when the nodes are scaled up again a new pod was running, so it should have triggered the case, shouldn't it?

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Aug 23, 2018

Yes, if a new pod started we need the logs to see why it didn't clean up.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Aug 23, 2018

The only option I have is try to reproduce the issue with the same configuration. If I manage to get something I will update the issue with the details.

@itskingori

This comment has been minimized.

Copy link

itskingori commented Aug 28, 2018

As @bboreham said node deletion does not clear IPs but reclaims them when a new node comes with same IP and it has no impact on performance.

@Raffo I just learnt that ☝️ from #3372 (comment). Key words, "same IP", which means with AWS autoscaling groups this will never happen. Could this be the problem?

I've pretty much subscribed to every issue on weave related to this problem and I've been using weave since around 1.8.x (see thread from #2797 (comment)). And we survive by cleaning up manually.

It's causing serious production issues for us (we scale quite a lot!) and I'm losing confidence (actually I've kinda given up today). Unfortunately we committed to weave early and changing CNI isn't trivial but possible (I asked on #kops-dev channel on kubernetes slack ... seriously considering this).

If you find a solution, please do share! 😅

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Aug 28, 2018

@itskingori please open a new issue with logs so we can diagnose what is causing you issues.

I have commented on "same IP" which seems to be a misunderstanding. Any Weave Net pod starting up should run the reclaim process, so just post the logs from that pod and it should give a clue.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Aug 28, 2018

@itskingori I will try as much as I can to reproduce it. Changing CNI provider is a possible alternative for me as well, but I'd love to stick with weave as the support has been amazing from the folks at weave and they deserve help in fixing that. Let's see what we can do 😄

@itskingori

This comment has been minimized.

Copy link

itskingori commented Aug 28, 2018

@Raffo ...

Changing CNI provider is a possible alternative for me as well

There's someone who's made a comment saying "A few people have tried doing this and not had success. We recommend just creating a new cluster", so I'd advice against it now. 😅

... but I'd love to stick with weave as the support has been amazing from the folks at weave and they deserve help in fixing that. Let's see what we can do

Yes they have. I would love to contribute but I'm out of my depth here ... plus networking is not my strong suite and the fact that this is hitting out production cluster is really making me sweat!


@bboreham I could but there are already issues that cover anything I have to say. Left a comment here because it's new and similar to #3372 which I'm tracking. There's also #3310 which is different and already filed. I've allocated some time to investigate this issue this week because of the severity and will add anything I find to the aforementioned issues.

@rade

This comment has been minimized.

Copy link
Member

rade commented Aug 28, 2018

@itskingori as @bboreham says, we really...

need the logs from a weave container starting after the point where it went wrong, to see why cleanup isn't working.

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Aug 28, 2018

there are already issues that cover anything I have to say

I ask for a separate issue for each report because it keeps the conversation focused. Similar-looking issues can be very different.

Yes there are people saying similar things but of all the thousands and thousands of instances running, nobody has posted a single log file of the weave container going wrong in the way described. We tried running up a new instance, scaling down and up, and it worked fine for us. We need to know what happened in your environment so we can look for the underlying cause.

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Aug 28, 2018

this can cause a clash of IPs.

@Raffo can you explain what you meant by this? You posted evidence of running out of available IPs, but nothing I can see as a “clash”.

@itskingori

This comment has been minimized.

Copy link

itskingori commented Aug 28, 2018

@rade Will give it another stab. Didn't have luck last time.

@bboreham Noted. Lemme try reproduce with a focus of getting logs.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Aug 29, 2018

@bboreham you are right, maybe they didn't clash, I will update the text.

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 4, 2018

we got bit by this problem today. We had it in the distant past-- but we thought it was resolved. We are using an ASG, so we dont have new nodes with the same IP

We rotate through our cluster every night, and delete each node one at a time. So all of the weave pods are restarted, and all of the nodes are restarted. It took a while to produce this-- definitely not just one night.

i suspect we might be rotating nodes often enough that they come online with ips that used to previously exist, but are now another MAC, same IP. I suspect this because i occasionally learn from ssh that the host key is wrong-- which means that this ip has been used before.

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 4, 2018

Below is a copy of IPAM status before we used rmpeer to clean things up. There is some interesting info to be seen there. For one, it is evident that our ip range is small enough that we see the same ip address for a host multiple times with a different MAC-- even within the below logs, for example these two:

a2:a2:b5:12:e0:67(ip-172-25-51-248.ec2.internal)      512 IPs (00.0% of total) - unreachable!
ca:73:0b:3f:88:f1(ip-172-25-51-248.ec2.internal)    16384 IPs (00.8% of total) - unreachable!

Full log ( which represents probably 3 weeks of running and recycling nodes )

[root@ip-172-25-19-231 log]# docker exec bf63ab3dbf6f /home/weave/weave --local status ipam
5a:b9:05:33:78:8f(ip-172-25-19-231.ec2.internal)   655367 IPs (31.3% of total) (7 active)
f6:c8:40:e4:39:01(ip-172-25-19-132.ec2.internal)      256 IPs (00.0% of total) - unreachable!
72:36:92:be:43:03(ip-172-25-51-58.ec2.internal)      128 IPs (00.0% of total) - unreachable!
0e:22:b8:1c:16:9b(ip-172-25-19-116.ec2.internal)      256 IPs (00.0% of total) - unreachable!
ee:45:c3:71:d1:3e(ip-172-25-19-83.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
1e:ab:d6:e8:12:e7(ip-172-25-51-20.ec2.internal)       96 IPs (00.0% of total) - unreachable!
96:d2:31:71:3d:af(ip-172-25-83-145.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
ca:c8:0c:61:68:82(ip-172-25-83-194.ec2.internal)    49152 IPs (02.3% of total) - unreachable!
8e:bb:55:a5:8e:20(ip-172-25-19-10.ec2.internal)    88595 IPs (04.2% of total) 
ca:7f:8a:1e:21:dc(ip-172-25-19-241.ec2.internal)     1057 IPs (00.1% of total) - unreachable!
9a:51:2d:db:21:0b(ip-172-25-83-206.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
6a:50:67:74:a1:24(ip-172-25-51-54.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
a2:8a:2c:70:a2:ab(ip-172-25-51-49.ec2.internal)      192 IPs (00.0% of total) - unreachable!
76:c7:ff:4d:61:de(ip-172-25-83-124.ec2.internal)       10 IPs (00.0% of total) - unreachable!
9a:de:2a:e3:4e:2f(ip-172-25-19-89.ec2.internal)       12 IPs (00.0% of total) - unreachable!
ca:d6:01:24:30:fe(ip-172-25-19-218.ec2.internal)      128 IPs (00.0% of total) - unreachable!
8e:f6:61:b1:95:0c(ip-172-25-19-140.ec2.internal)       48 IPs (00.0% of total) - unreachable!
0e:fe:46:cc:75:51(ip-172-25-51-207.ec2.internal)       74 IPs (00.0% of total) - unreachable!
1e:09:d2:f9:ee:4f(ip-172-25-83-33.ec2.internal)    49152 IPs (02.3% of total) - unreachable!
de:f1:7e:87:c1:06(ip-172-25-83-140.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
1a:fb:22:64:df:9d(ip-172-25-51-132.ec2.internal)     1024 IPs (00.0% of total) - unreachable!
46:e1:e5:dc:02:14(ip-172-25-51-31.ec2.internal)       32 IPs (00.0% of total) - unreachable!
a2:3d:ac:94:85:2f(ip-172-25-19-201.ec2.internal)        2 IPs (00.0% of total) - unreachable!
12:bf:5b:f2:3d:66(ip-172-25-19-67.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
02:3c:b7:fe:52:f0(ip-172-25-51-207.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
4a:d4:11:e8:65:1e(ip-172-25-19-166.ec2.internal)       32 IPs (00.0% of total) - unreachable!
c2:ff:5a:b5:dd:9b(ip-172-25-51-24.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
ea:66:b7:e3:f0:fb(ip-172-25-19-147.ec2.internal)      256 IPs (00.0% of total) - unreachable!
d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)    43840 IPs (02.1% of total) 
7e:06:18:eb:2e:98(ip-172-25-51-181.ec2.internal)        3 IPs (00.0% of total) - unreachable!
e2:ad:de:80:63:5f(ip-172-25-19-65.ec2.internal)    81920 IPs (03.9% of total) - unreachable!
62:2d:1e:bb:4c:8a(ip-172-25-51-232.ec2.internal)       24 IPs (00.0% of total) - unreachable!
d6:ff:ab:76:d8:10(ip-172-25-19-22.ec2.internal)     1024 IPs (00.0% of total) - unreachable!
22:65:b2:27:a8:0d(ip-172-25-51-74.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
1e:de:e1:d0:30:6b(ip-172-25-19-74.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
a6:cb:93:2e:dc:38(ip-172-25-83-112.ec2.internal)     1024 IPs (00.0% of total) - unreachable!
7a:3f:f4:50:41:97(ip-172-25-83-45.ec2.internal)        3 IPs (00.0% of total) - unreachable!
7a:e7:98:4d:5b:1d(ip-172-25-83-81.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
06:39:2a:5f:ce:5b(ip-172-25-83-93.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
fa:36:7a:11:ea:b5(ip-172-25-51-68.ec2.internal)    12288 IPs (00.6% of total) - unreachable!
16:74:d7:8d:66:19(ip-172-25-19-71.ec2.internal)      512 IPs (00.0% of total) - unreachable!
ba:69:a0:bb:b3:68(ip-172-25-51-34.ec2.internal)      768 IPs (00.0% of total) - unreachable!
be:f1:cf:b2:c0:bb(ip-172-25-51-91.ec2.internal)       24 IPs (00.0% of total) - unreachable!
d6:a7:e3:c4:3b:22(ip-172-25-83-244.ec2.internal)        4 IPs (00.0% of total) - unreachable!
22:3d:37:c3:40:cc(ip-172-25-19-210.ec2.internal)      512 IPs (00.0% of total) - unreachable!
b2:2c:8e:1a:c4:4d(ip-172-25-19-237.ec2.internal)      256 IPs (00.0% of total) - unreachable!
de:a1:bd:9f:d8:d9(ip-172-25-51-226.ec2.internal)   131072 IPs (06.2% of total) - unreachable!
d6:92:e0:f5:a0:0f(ip-172-25-51-253.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
32:10:d9:59:73:c8(ip-172-25-83-128.ec2.internal)       24 IPs (00.0% of total) - unreachable!
be:cb:b2:bc:15:39(ip-172-25-51-198.ec2.internal)        8 IPs (00.0% of total) - unreachable!
12:38:9a:2d:05:1a(ip-172-25-83-190.ec2.internal)    12288 IPs (00.6% of total) - unreachable!
be:6f:2a:a2:78:4d(ip-172-25-51-148.ec2.internal)      128 IPs (00.0% of total) - unreachable!
86:bb:9c:23:92:8c(ip-172-25-83-133.ec2.internal)       48 IPs (00.0% of total) - unreachable!
36:a9:e3:3e:52:ee(ip-172-25-19-80.ec2.internal)       32 IPs (00.0% of total) - unreachable!
2a:1d:56:4d:8d:dd(ip-172-25-83-139.ec2.internal)        2 IPs (00.0% of total) 
e6:0d:ae:cf:7e:d6(ip-172-25-19-59.ec2.internal)       16 IPs (00.0% of total) - unreachable!
a2:a2:b5:12:e0:67(ip-172-25-51-248.ec2.internal)      512 IPs (00.0% of total) - unreachable!
3e:2b:4d:9c:0e:00(ip-172-25-83-17.ec2.internal)        2 IPs (00.0% of total) - unreachable!
ca:73:0b:3f:88:f1(ip-172-25-51-248.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
b2:f3:c5:99:7d:ce(ip-172-25-19-19.ec2.internal)      128 IPs (00.0% of total) - unreachable!
32:d4:54:1c:6f:5e(ip-172-25-19-204.ec2.internal)      768 IPs (00.0% of total) - unreachable!
9a:a0:fe:6c:cc:e2(ip-172-25-51-95.ec2.internal)      143 IPs (00.0% of total) - unreachable!
ea:5f:c7:32:c3:e8(ip-172-25-83-160.ec2.internal)    13832 IPs (00.7% of total) 
4a:8a:50:62:cf:74(ip-172-25-83-209.ec2.internal)     3072 IPs (00.1% of total) - unreachable!
56:03:f3:5b:5e:e3(ip-172-25-51-16.ec2.internal)       32 IPs (00.0% of total) - unreachable!
22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)    79328 IPs (03.8% of total) 
4a:57:0e:39:0c:3f(ip-172-25-83-156.ec2.internal)   196608 IPs (09.4% of total) - unreachable!
aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)    88837 IPs (04.2% of total) 
e2:5e:06:09:c1:8b(ip-172-25-51-19.ec2.internal)       32 IPs (00.0% of total) - unreachable!
b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)    43966 IPs (02.1% of total) 
62:6f:b2:b7:9d:e4(ip-172-25-83-33.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
3a:eb:40:62:53:05(ip-172-25-83-52.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
06:4e:6f:7f:f0:0f(ip-172-25-83-217.ec2.internal)      384 IPs (00.0% of total) - unreachable!
06:21:a0:a5:0b:a2(ip-172-25-83-131.ec2.internal)     1024 IPs (00.0% of total) - unreachable!
62:ae:1c:4c:11:14(ip-172-25-83-6.ec2.internal)       64 IPs (00.0% of total) - unreachable!
c6:7c:6b:b1:33:ba(ip-172-25-51-163.ec2.internal)      512 IPs (00.0% of total) - unreachable!
36:c4:4c:c2:8f:5e(ip-172-25-83-54.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
9e:5c:82:8f:2d:39(ip-172-25-51-124.ec2.internal)    49152 IPs (02.3% of total) - unreachable!
6a:a6:87:8f:0d:92(ip-172-25-51-89.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
9a:c0:7e:fd:a0:34(ip-172-25-19-93.ec2.internal)      277 IPs (00.0% of total) - unreachable!
6e:01:90:44:7d:6a(ip-172-25-19-126.ec2.internal)     3072 IPs (00.1% of total) - unreachable!
4e:be:aa:2e:e0:0b(ip-172-25-19-134.ec2.internal)     1024 IPs (00.0% of total) - unreachable!
9a:eb:6d:d5:ab:57(ip-172-25-83-248.ec2.internal)      529 IPs (00.0% of total) - unreachable!
86:f0:48:88:5d:b5(ip-172-25-83-178.ec2.internal)      512 IPs (00.0% of total) - unreachable!
ce:16:a3:55:b6:d3(ip-172-25-51-58.ec2.internal)       32 IPs (00.0% of total) - unreachable!
7a:b7:b6:76:9c:a8(ip-172-25-51-34.ec2.internal)    32769 IPs (01.6% of total) 
7a:2e:ab:24:f3:05(ip-172-25-51-5.ec2.internal)       48 IPs (00.0% of total) - unreachable!
ee:c5:88:ee:6d:ad(ip-172-25-19-103.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
56:99:ef:c2:35:65(ip-172-25-19-196.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
1e:da:eb:19:d6:2c(ip-172-25-83-194.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
6e:28:93:43:3f:2c(ip-172-25-19-14.ec2.internal)        5 IPs (00.0% of total) - unreachable!
ae:60:7e:a9:e1:1f(ip-172-25-51-174.ec2.internal)       18 IPs (00.0% of total) - unreachable!
ee:bf:6c:33:e1:15(ip-172-25-83-79.ec2.internal)      256 IPs (00.0% of total) - unreachable!
96:4f:0f:ce:0b:9a(ip-172-25-19-238.ec2.internal)        8 IPs (00.0% of total) - unreachable!
ea:a3:9e:bc:91:ba(ip-172-25-51-106.ec2.internal)   196608 IPs (09.4% of total) - unreachable!
62:0d:3c:55:24:bc(ip-172-25-19-208.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
ae:13:12:17:70:21(ip-172-25-51-8.ec2.internal)    32768 IPs (01.6% of total) - unreachable!

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 4, 2018

@bboreham given that we cycle through our cluster frequently, and ran into this issue, we would like to implement a workaround until the issue is resolved. My understanding from reading most of the issues is that part of the difficulty in applying a fix is that it is hard to know when a node has gone permanently and when it is temporary.

In our case, we control the process of terminating nodes, so we know when it is. Our script drains and then terminates nodes in a predictable order.

If we could do something at the time we terminate a node, what would the 'right thing' to do be?

would it work to simply run 'weave reset' on each node after we finish restarting them all?

Or would it be better to randomly select a node and run 'weave rmpeer <hostname we just removed> on one of them?

One of our annoying details is that we can no longer run kubectl exec on the weave pods-- they are using hostNetwork true, so we can't exec on them remotely.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 4, 2018

Thanks for posting your update regarding this issue! In the meantime I have spin up a test that should reproduce a similar problem (I'm just spinning up and down nodes in a loop, let's see if it leads to the same issues in a reproducible way).

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 5, 2018

@Raffo sounds good.

Let's suppose that we can duplicate it... What is the fix then?

It seems that even when we know it happens, the fix isn't clear. How do you know how long to wait for a node before you give up on it?

For the record, in our case the suggestion I think I read above, one week, would work. It took us three weeks to get this to happen.

Could we go with a configurable how long to wait value, after which unreachable nodes are removed, with a default value of one week?

@brb

This comment has been minimized.

Copy link
Contributor

brb commented Sep 5, 2018

@dcowden

Below is a copy of IPAM status before we used rmpeer to clean things up. There is some interesting info to be seen there. For one, it is evident that our ip range is small enough that we see the same ip address for a host multiple times with a different MAC-- even within the below logs, for example these two:

(Correction: it's a peer name, not MAC).

The IP addr re-use should definitely cause problem. The IPAM reclaimer identifies nodes by their (host)name (https://github.com/weaveworks/weave/blob/v2.4.0/prog/kube-utils/main.go#L114), so a dead node cannot be rmpeer'd if there is a running node with the same IP addr. We should address this.

To better understand what happens in your case, I need full logs (not ipam status) of the weave container which has started recently.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 5, 2018

Well, in only one night running I managed to replicate the issue.
The strategy is the following:

  1. I create a new InstanceGroup in kops with a dedicated AWS ASG.
  2. Scale to 3 instances
  3. Sleep for 5 minutes
  4. Scale to 0
  5. Sleep 5
  6. GOTO 2 :-)

As requested by @bboreham and @brb , this is the log of a weave container (v 2.3 as stated in this issue) starting after the problem started happening:

INFO: 2018/09/05 08:02:48.919554 Command line options: map[metrics-addr:0.0.0.0:6782 mtu:8912 docker-api: ipalloc-range:100.96.0.0/11 http-addr:127.0.0.1:6784 nickname:ip-10-11-97-47.us-west-1.compute.internal port:6783 datapath:datapath host-root:/host conn-limit:100 name:86:33:d4:18:4c:a1 ipalloc-init:consensus=7 no-dns:true db-prefix:/weavedb/weave-net expect-npc:true]
INFO: 2018/09/05 08:02:48.919603 weave  2.3.0
INFO: 2018/09/05 08:02:49.515667 Bridge type is bridged_fastdp
INFO: 2018/09/05 08:02:49.515683 Communication between peers is unencrypted.
INFO: 2018/09/05 08:02:49.517684 Our name is 86:33:d4:18:4c:a1(ip-10-11-97-47.us-west-1.compute.internal)
INFO: 2018/09/05 08:02:49.517722 Launch detected - using supplied peer list: [10.11.46.213 10.11.47.229 10.11.52.39 10.11.68.17 10.11.79.169 10.11.97.215 10.11.97.47]
INFO: 2018/09/05 08:02:49.517749 Checking for pre-existing addresses on weave bridge
INFO: 2018/09/05 08:02:49.520640 [allocator 86:33:d4:18:4c:a1] No valid persisted data
INFO: 2018/09/05 08:02:49.524650 [allocator 86:33:d4:18:4c:a1] Initialising via deferred consensus
INFO: 2018/09/05 08:02:49.524677 Sniffing traffic on datapath (via ODP)
INFO: 2018/09/05 08:02:49.533770 ->[10.11.68.17:6783] attempting connection
INFO: 2018/09/05 08:02:49.533869 ->[10.11.79.169:6783] attempting connection
INFO: 2018/09/05 08:02:49.533956 ->[10.11.97.215:6783] attempting connection
INFO: 2018/09/05 08:02:49.534011 ->[10.11.97.47:6783] attempting connection
INFO: 2018/09/05 08:02:49.534202 ->[10.11.46.213:6783] attempting connection
INFO: 2018/09/05 08:02:49.534249 ->[10.11.47.229:6783] attempting connection
INFO: 2018/09/05 08:02:49.534295 ->[10.11.52.39:6783] attempting connection
INFO: 2018/09/05 08:02:49.534698 ->[10.11.97.47:59974] connection accepted
INFO: 2018/09/05 08:02:49.537803 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2018/09/05 08:02:49.537987 Listening for metrics requests on 0.0.0.0:6782
INFO: 2018/09/05 08:02:49.539920 ->[10.11.79.169:6783] error during connection attempt: dial tcp4 :0->10.11.79.169:6783: connect: connection refused
INFO: 2018/09/05 08:02:49.540097 ->[10.11.97.47:59974|86:33:d4:18:4c:a1(ip-10-11-97-47.us-west-1.compute.internal)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/09/05 08:02:49.540356 ->[10.11.68.17:6783|b2:41:e5:3b:fe:89(ip-10-11-68-17.us-west-1.compute.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 08:02:49.540451 overlay_switch ->[b2:41:e5:3b:fe:89(ip-10-11-68-17.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.540476 ->[10.11.68.17:6783|b2:41:e5:3b:fe:89(ip-10-11-68-17.us-west-1.compute.internal)]: connection added (new peer)
INFO: 2018/09/05 08:02:49.541489 ->[10.11.52.39:6783|ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 08:02:49.541547 overlay_switch ->[ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.541562 ->[10.11.52.39:6783|ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)]: connection added (new peer)
INFO: 2018/09/05 08:02:49.542128 ->[10.11.97.215:6783|6e:ac:1b:5a:8d:b3(ip-10-11-97-215.us-west-1.compute.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 08:02:49.542394 overlay_switch ->[6e:ac:1b:5a:8d:b3(ip-10-11-97-215.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.542703 ->[10.11.97.215:6783|6e:ac:1b:5a:8d:b3(ip-10-11-97-215.us-west-1.compute.internal)]: connection added (new peer)
INFO: 2018/09/05 08:02:49.543293 ->[10.11.97.47:6783|86:33:d4:18:4c:a1(ip-10-11-97-47.us-west-1.compute.internal)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/09/05 08:02:49.550581 ->[10.11.46.213:6783|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 08:02:49.550634 overlay_switch ->[9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.550668 ->[10.11.46.213:6783|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection added (new peer)
INFO: 2018/09/05 08:02:49.557076 ->[10.11.47.229:6783|f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 08:02:49.557134 overlay_switch ->[f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.557156 ->[10.11.47.229:6783|f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)]: connection added (new peer)
INFO: 2018/09/05 08:02:49.559067 ->[10.11.46.213:60536] connection accepted
INFO: 2018/09/05 08:02:49.560615 ->[10.11.46.213:6783|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection fully established
INFO: 2018/09/05 08:02:49.560907 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2018/09/05 08:02:49.561967 ->[10.11.97.215:6783|6e:ac:1b:5a:8d:b3(ip-10-11-97-215.us-west-1.compute.internal)]: connection fully established
INFO: 2018/09/05 08:02:49.562209 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2018/09/05 08:02:49.562276 overlay_switch ->[b2:41:e5:3b:fe:89(ip-10-11-68-17.us-west-1.compute.internal)] using sleeve
INFO: 2018/09/05 08:02:49.562290 ->[10.11.68.17:6783|b2:41:e5:3b:fe:89(ip-10-11-68-17.us-west-1.compute.internal)]: connection fully established
INFO: 2018/09/05 08:02:49.562383 overlay_switch ->[b2:41:e5:3b:fe:89(ip-10-11-68-17.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.562590 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2018/09/05 08:02:49.562638 overlay_switch ->[ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)] using sleeve
INFO: 2018/09/05 08:02:49.562651 ->[10.11.52.39:6783|ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)]: connection fully established
INFO: 2018/09/05 08:02:49.562773 overlay_switch ->[ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.563460 ->[10.11.46.213:60536|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 08:02:49.563490 overlay_switch ->[9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.563506 ->[10.11.46.213:6783|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection deleted
INFO: 2018/09/05 08:02:49.563571 ->[10.11.46.213:60536|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection added
INFO: 2018/09/05 08:02:49.564337 ->[10.11.46.213:6783|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection shutting down due to error: Multiple connections to 9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal) added to 86:33:d4:18:4c:a1(ip-10-11-97-47.us-west-1.compute.internal)
INFO: 2018/09/05 08:02:49.564997 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2018/09/05 08:02:49.565045 overlay_switch ->[f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)] using sleeve
INFO: 2018/09/05 08:02:49.565057 ->[10.11.47.229:6783|f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)]: connection fully established
INFO: 2018/09/05 08:02:49.565436 sleeve ->[10.11.68.17:6783|b2:41:e5:3b:fe:89(ip-10-11-68-17.us-west-1.compute.internal)]: Effective MTU verified at 8939
INFO: 2018/09/05 08:02:49.565494 sleeve ->[10.11.52.39:6783|ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)]: Effective MTU verified at 8939
INFO: 2018/09/05 08:02:49.566510 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2018/09/05 08:02:49.566997 sleeve ->[10.11.97.215:6783|6e:ac:1b:5a:8d:b3(ip-10-11-97-215.us-west-1.compute.internal)]: Effective MTU verified at 8939
INFO: 2018/09/05 08:02:49.567067 overlay_switch ->[f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:02:49.567086 sleeve ->[10.11.47.229:6783|f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)]: Effective MTU verified at 8939
INFO: 2018/09/05 08:02:49.570088 ->[10.11.46.213:60536|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: connection fully established
INFO: 2018/09/05 08:02:49.572437 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2018/09/05 08:02:49.573482 sleeve ->[10.11.46.213:6783|9a:6b:df:34:bd:5a(ip-10-11-46-213.us-west-1.compute.internal)]: Effective MTU verified at 8939
INFO: 2018/09/05 08:02:49.783414 Discovered remote MAC 10:4f:d9:ed:76:11 at f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)
INFO: 2018/09/05 08:02:49.990086 Weave version 2.4.0 is available; please update at https://github.com/weaveworks/weave/releases/download/v2.4.0/weave
INFO: 2018/09/05 08:02:50.003952 [kube-peers] Added myself to peer list &{[{9a:6b:df:34:bd:5a ip-10-11-46-213.us-west-1.compute.internal} {ba:57:58:4e:78:77 ip-10-11-52-39.us-west-1.compute.internal} {b2:41:e5:3b:fe:89 ip-10-11-68-17.us-west-1.compute.internal} {6e:ac:1b:5a:8d:b3 ip-10-11-97-215.us-west-1.compute.internal} {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal} {f2:5d:42:1d:63:af ip-10-11-47-229.us-west-1.compute.internal} {86:33:d4:18:4c:a1 ip-10-11-97-47.us-west-1.compute.internal}]}
DEBU: 2018/09/05 08:02:50.006160 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.006180 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.006188 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.008243 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.008267 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.008275 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.010419 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.010436 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.010443 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.012573 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.012591 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.012597 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.014636 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.014653 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.014660 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.016674 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.016691 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.016697 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.018672 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.018692 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.018699 [kube-peers] Existing annotation f2:5d:42:1d:63:af
INFO: 2018/09/05 08:02:50.071304 Discovered remote MAC f2:5d:42:1d:63:af at f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)
DEBU: 2018/09/05 08:02:50.184334 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.184359 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.184369 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.384208 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.384234 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.384241 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.584562 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.584585 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.584594 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.784623 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.784649 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.784657 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:50.984644 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:50.984670 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:50.984677 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:51.185039 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:51.185080 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:51.185088 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:51.384673 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:51.384698 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:51.384707 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:51.584639 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:51.584663 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:51.584671 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:51.784675 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:51.784700 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:51.784708 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:51.984690 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:51.984717 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:51.984725 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:52.184901 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:52.184928 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:52.184937 [kube-peers] Existing annotation f2:5d:42:1d:63:af
INFO: 2018/09/05 08:02:52.290641 Discovered remote MAC 4e:64:58:2d:d5:d8 at f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)
DEBU: 2018/09/05 08:02:52.384851 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:52.384877 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:52.384892 [kube-peers] Existing annotation f2:5d:42:1d:63:af
INFO: 2018/09/05 08:02:52.507176 ->[10.11.79.169:6783] attempting connection
INFO: 2018/09/05 08:02:52.508018 ->[10.11.79.169:6783] error during connection attempt: dial tcp4 :0->10.11.79.169:6783: connect: connection refused
DEBU: 2018/09/05 08:02:52.584785 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:52.584812 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:52.584820 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:52.784933 [kube-peers] Nodes that have disappeared: map[ip-10-11-117-137.us-west-1.compute.internal:{fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}]
DEBU: 2018/09/05 08:02:52.784967 [kube-peers] Preparing to remove disappeared peer {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal}
DEBU: 2018/09/05 08:02:52.784977 [kube-peers] Existing annotation f2:5d:42:1d:63:af
DEBU: 2018/09/05 08:02:52.984557 [kube-peers] Nodes that have disappeared: map[]
100.126.128.0
INFO: 2018/09/05 08:02:53.039343 Discovered remote MAC 3e:8a:91:04:e6:89 at f2:5d:42:1d:63:af(ip-10-11-47-229.us-west-1.compute.internal)
INFO: 2018/09/05 08:02:55.108177 ->[10.11.79.169:6783] attempting connection
INFO: 2018/09/05 08:02:55.109028 ->[10.11.79.169:6783] error during connection attempt: dial tcp4 :0->10.11.79.169:6783: connect: connection refused
INFO: 2018/09/05 08:02:59.399220 ->[10.11.79.169:6783] attempting connection
INFO: 2018/09/05 08:02:59.400072 ->[10.11.79.169:6783] error during connection attempt: dial tcp4 :0->10.11.79.169:6783: connect: connection refused
INFO: 2018/09/05 08:03:02.741845 ->[10.11.79.169:58305] connection accepted
INFO: 2018/09/05 08:03:02.759238 ->[10.11.79.169:58305|ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 08:03:02.759460 overlay_switch ->[ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)] using fastdp
INFO: 2018/09/05 08:03:02.759483 ->[10.11.79.169:58305|ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)]: connection added (new peer)
INFO: 2018/09/05 08:03:02.762420 ->[10.11.79.169:58305|ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)]: connection fully established
INFO: 2018/09/05 08:03:02.772575 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2018/09/05 08:03:02.783507 sleeve ->[10.11.79.169:6783|ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)]: Effective MTU verified at 8939
INFO: 2018/09/05 08:03:02.825050 Discovered remote MAC 92:9a:e9:3e:d9:db at ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:03.061251 Discovered remote MAC ca:36:f4:2c:a4:b2 at ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:03.233104 Discovered remote MAC e6:41:fa:8d:95:3e at ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:07.522112 Discovered remote MAC 52:61:aa:6c:d4:ab at ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:07.524249 Discovered remote MAC 76:c9:e6:4e:01:2e at ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:23.178467 Discovered remote MAC ea:ec:78:e4:48:83 at ca:36:f4:2c:a4:b2(ip-10-11-79-169.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:43.624579 Discovered remote MAC ca:a4:bc:bb:a5:e3 at ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:43.641409 Discovered remote MAC 82:1b:a3:2c:46:8a at ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:43.688159 Discovered remote MAC 76:86:b2:a1:d2:eb at ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)
INFO: 2018/09/05 08:03:43.714282 Discovered remote MAC 8a:e7:ed:59:c0:d7 at ba:57:58:4e:78:77(ip-10-11-52-39.us-west-1.compute.internal)

From my understanding, and I apologize if I am wrong, any protocol that will deal with peer discovery will have to assume at some point that the peers are gone and it's totally normal on a cloud scenario that IP addresses will be reused so this problem will happen for sure, it is only a matter of time. I guess a timeout of nodes "being gone" could already address this, but to be on the safe side we have to deal with the conflicts in general. WDYT? /cc @brb

@brb

This comment has been minimized.

Copy link
Contributor

brb commented Sep 5, 2018

@Raffo

Well, in only one night running I managed to replicate the issue.

Thanks for trying to replicate, but which issue you refer to? The log looks healthy.

Also, the multiple DEBU: 2018/09/05 08:02:50.984677 [kube-peers] Existing annotation f2:5d:42:1d:63:af have been addressed in #3317 (released in 2.4.0, so please update).

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 5, 2018

@brb as I wrote, I have exactly the problem in the main issue where many nodes are unreachable even if the log looks healthy :

weave-net-2wgzd   2/2       Running   1          3m        10.11.87.176   ip-10-11-87-176.us-west-1.compute.internal
f6:98:36:eb:ea:01(ip-10-11-87-176.us-west-1.compute.internal)    16384 IPs (00.8% of total) (2 active)
ba:ad:4d:aa:2d:02(ip-10-11-107-80.us-west-1.compute.internal)   393216 IPs (18.8% of total) - unreachable!
ee:39:1a:28:c0:03(ip-10-11-32-63.us-west-1.compute.internal)   393216 IPs (18.8% of total) - unreachable!
ba:57:58:4e:78:04(ip-10-11-52-39.us-west-1.compute.internal)    49152 IPs (02.3% of total)
5a:bc:5c:89:13:05(ip-10-11-99-14.us-west-1.compute.internal)   196608 IPs (09.4% of total)
b2:a5:7c:c0:18:06(ip-10-11-39-77.us-west-1.compute.internal)    49152 IPs (02.3% of total)
f6:19:aa:63:40:07(ip-10-11-82-112.us-west-1.compute.internal)    16384 IPs (00.8% of total) - unreachable!
72:7a:d2:c8:8e:08(ip-10-11-34-180.us-west-1.compute.internal)   524288 IPs (25.0% of total) - unreachable!
9a:6b:df:34:bd:09(ip-10-11-46-213.us-west-1.compute.internal)    32768 IPs (01.6% of total)
6e:ac:1b:5a:8d:10(ip-10-11-97-215.us-west-1.compute.internal)    65536 IPs (03.1% of total)
26:16:09:8d:8e:11(ip-10-11-112-170.us-west-1.compute.internal)   262144 IPs (12.5% of total) - unreachable!
b2:41:e5:3b:fe:12(ip-10-11-68-17.us-west-1.compute.internal)    65536 IPs (03.1% of total)
86:33:d4:18:4c:13(ip-10-11-97-47.us-west-1.compute.internal)    32768 IPs (01.6% of total) - unreachable!
@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 5, 2018

None of those are in the peer list, so they won't get cleaned up now.

INFO: 2018/09/05 08:02:50.003952 [kube-peers] Added myself to peer list &{[{9a:6b:df:34:bd:5a ip-10-11-46-213.us-west-1.compute.internal} {ba:57:58:4e:78:77 ip-10-11-52-39.us-west-1.compute.internal} {b2:41:e5:3b:fe:89 ip-10-11-68-17.us-west-1.compute.internal} {6e:ac:1b:5a:8d:b3 ip-10-11-97-215.us-west-1.compute.internal} {fa:03:c1:19:bd:fb ip-10-11-117-137.us-west-1.compute.internal} {f2:5d:42:1d:63:af ip-10-11-47-229.us-west-1.compute.internal} {86:33:d4:18:4c:a1 ip-10-11-97-47.us-west-1.compute.internal}]}

Do you have logs from earlier? Presumably something went wrong but it isn't in that log.

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 5, 2018

Hi @brb
Our cluster is kubernetes 1.10.3, built on kops 1.10, with weave 2.4.0.
It consists of 3 masters and 6 workers on aws, evenly distributed across 3 AZs.

The logs below are from our development cluster, which has an artificially accelerated maintenance schedule ( nightly ) to try to make these kinds of things happen more frequently ( its working :) )

Overnight, a script rolls through all nodes, and drains and terminates them one by one. Yesterday, we cleared out all of the unreachable nodes. The output below is from a healthy cluster, after 1 'round' of upgrades. In practice it takes about a week or two before we get a broken cluster due to the accumulation of unreachable peers.

My (unconfirmed) suspicion is that we have a broken cluster as soon as an ip address gets re-used.

It would seem reasonable to me to automatically remove old peers when a new peer is discovered having a new host name but different MAC. its possible that would fix our case, despite all of the unreachable peers.

[root@ip-172-25-19-155 ~]# docker exec d5e45b0be524 /home/weave/weave --local status ipam
62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)   483851 IPs (23.1% of total) (9 active)
6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)   131072 IPs (06.2% of total) 
5a:b9:05:33:78:8f(ip-172-25-19-231.ec2.internal)   655367 IPs (31.3% of total) - unreachable!
7a:b7:b6:76:9c:a8(ip-172-25-51-34.ec2.internal)    32769 IPs (01.6% of total) - unreachable!
b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)    43966 IPs (02.1% of total) - unreachable!
46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)    79328 IPs (03.8% of total) 
d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)    43840 IPs (02.1% of total) 
aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)   262144 IPs (12.5% of total) 
c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)    88837 IPs (04.2% of total) 
ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)    13832 IPs (00.7% of total) 
2a:1d:56:4d:8d:dd(ip-172-25-83-139.ec2.internal)        2 IPs (00.0% of total) - unreachable!
ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)   131072 IPs (06.2% of total) 
d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)   131072 IPs (06.2% of total) 

[root@ip-172-25-19-155 ~]# docker logs d5e45b0be524
INFO: 2018/09/05 03:42:56.445498 Command line options: map[conn-limit:100 expect-npc:true ipalloc-init:consensus=9 port:6783 datapath:datapath docker-api: http-addr:127.0.0.1:6784 ipalloc-range:100.96.0.0/11 nickname:ip-172-25-19-155.ec2.internal db-prefix:/weavedb/weave-net host-root:/host mtu:8912 name:62:b1:3d:d5:2e:5e metrics-addr:0.0.0.0:6782 no-dns:true]
INFO: 2018/09/05 03:42:56.445582 weave  2.4.0
INFO: 2018/09/05 03:42:56.455354 failed to create weave-test-comment2f9f4907; disabling comment support
INFO: 2018/09/05 03:42:56.867472 Bridge type is bridged_fastdp
INFO: 2018/09/05 03:42:56.867494 Communication between peers via untrusted networks is encrypted.
INFO: 2018/09/05 03:42:57.034008 Our name is 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 03:42:57.034057 Launch detected - using supplied peer list: [172.25.19.155 172.25.19.165 172.25.19.192 172.25.51.190 172.25.51.245 172.25.51.92 172.25.83.103 172.25.83.156 172.25.83.167]
INFO: 2018/09/05 03:42:57.034081 Checking for pre-existing addresses on weave bridge
INFO: 2018/09/05 03:42:57.057642 [allocator 62:b1:3d:d5:2e:5e] No valid persisted data
INFO: 2018/09/05 03:42:57.259855 [allocator 62:b1:3d:d5:2e:5e] Initialising via deferred consensus
INFO: 2018/09/05 03:42:57.259900 Sniffing traffic on datapath (via ODP)
INFO: 2018/09/05 03:42:57.260530 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 03:42:57.260705 ->[172.25.51.92:6783] attempting connection
INFO: 2018/09/05 03:42:57.260770 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 03:42:57.260864 ->[172.25.19.192:6783] attempting connection
INFO: 2018/09/05 03:42:57.260927 ->[172.25.51.245:6783] attempting connection
INFO: 2018/09/05 03:42:57.260978 ->[172.25.19.155:6783] attempting connection
INFO: 2018/09/05 03:42:57.261057 ->[172.25.83.167:6783] attempting connection
INFO: 2018/09/05 03:42:57.261111 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 03:42:57.261183 ->[172.25.19.155:43843] connection accepted
INFO: 2018/09/05 03:42:57.261285 ->[172.25.83.156:6783] attempting connection
INFO: 2018/09/05 03:42:57.264152 ->[172.25.83.156:6783|aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.264225 overlay_switch ->[aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.264291 ->[172.25.83.156:6783|aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.264469 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)
INFO: 2018/09/05 03:42:57.265098 ipsec: InitSALocal: 172.25.83.156 -> 172.25.19.155 :6784 0x5fc70829
INFO: 2018/09/05 03:42:57.265328 ->[172.25.19.155:6783|62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/09/05 03:42:57.265585 ->[172.25.19.165:6783|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.265620 overlay_switch ->[22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.265638 ->[172.25.19.165:6783|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.265700 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:42:57.266568 ->[172.25.83.103:6783|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.266630 overlay_switch ->[aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.266647 ->[172.25.83.103:6783|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.266699 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:42:57.267055 ->[172.25.83.103:52465] connection accepted
INFO: 2018/09/05 03:42:57.267433 ->[172.25.51.92:6783|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.267466 overlay_switch ->[d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.267480 ->[172.25.51.92:6783|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.267535 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)
INFO: 2018/09/05 03:42:57.267642 ->[172.25.19.155:43843|62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/09/05 03:42:57.267742 ->[172.25.51.245:6783|d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.267777 overlay_switch ->[d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.267837 ->[172.25.51.245:6783|d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.267914 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:42:57.268011 ->[172.25.19.192:6783|ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.268048 overlay_switch ->[ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.268062 ->[172.25.19.192:6783|ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.268111 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)
INFO: 2018/09/05 03:42:57.269072 ->[172.25.83.167:6783|ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.269106 overlay_switch ->[ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.269123 ->[172.25.83.167:6783|ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.269174 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:42:57.269964 ->[172.25.19.165:42024] connection accepted
INFO: 2018/09/05 03:42:57.270008 ->[172.25.51.190:43183] connection accepted
INFO: 2018/09/05 03:42:57.270035 ->[172.25.51.92:60214] connection accepted
INFO: 2018/09/05 03:42:57.270069 ->[172.25.51.245:33251] connection accepted
INFO: 2018/09/05 03:42:57.270850 ->[172.25.83.103:52465|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.270902 overlay_switch ->[aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.270927 ->[172.25.83.103:52465|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: connection shutting down due to error: Multiple connections to aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 03:42:57.271112 ->[172.25.51.190:6783|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.271155 overlay_switch ->[b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.271170 ->[172.25.51.190:6783|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.271223 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:42:57.272000 ->[172.25.19.165:42024|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.272033 overlay_switch ->[22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.272051 ->[172.25.19.165:6783|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection deleted
INFO: 2018/09/05 03:42:57.272101 ->[172.25.19.165:42024|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.272166 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:42:57.272995 ->[172.25.51.92:60214|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.273038 overlay_switch ->[d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.273054 ->[172.25.51.92:6783|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: connection deleted
INFO: 2018/09/05 03:42:57.273106 ->[172.25.51.92:60214|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.273163 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)
INFO: 2018/09/05 03:42:57.274683 ->[172.25.51.245:33251|d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.274728 overlay_switch ->[d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.274753 ->[172.25.51.245:33251|d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)]: connection shutting down due to error: Multiple connections to d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 03:42:57.275152 ->[172.25.51.190:43183|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 03:42:57.275190 overlay_switch ->[b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.275205 ->[172.25.51.190:6783|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection deleted
INFO: 2018/09/05 03:42:57.275253 ->[172.25.51.190:43183|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 03:42:57.275300 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:42:57.323199 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2018/09/05 03:42:57.323405 Listening for metrics requests on 0.0.0.0:6782
INFO: 2018/09/05 03:42:57.493933 ipsec: InitSALocal: 172.25.19.165 -> 172.25.19.155 :6784 0x5038c53c
ERRO: 2018/09/05 03:42:57.502674 fastdp ->[172.25.19.165:6784|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: ipsec init SA local failed: send InitSARemote: write tcp4 172.25.19.155:59085->172.25.19.165:6783: write: connection reset by peer
INFO: 2018/09/05 03:42:57.502708 ->[172.25.19.165:6783|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection shutting down due to error: Multiple connections to 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 03:42:57.502852 ipsec: InitSALocal: 172.25.83.103 -> 172.25.19.155 :6784 0x96a99a41
INFO: 2018/09/05 03:42:57.503080 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:42:57.503114 overlay_switch ->[22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)] fastdp send InitSARemote: write tcp4 172.25.19.155:59085->172.25.19.165:6783: write: connection reset by peer
INFO: 2018/09/05 03:42:57.508052 ipsec: InitSALocal: 172.25.51.92 -> 172.25.19.155 :6784 0x8cdc6e29
ERRO: 2018/09/05 03:42:57.520953 fastdp ->[172.25.51.92:6784|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: ipsec init SA local failed: send InitSARemote: write tcp4 172.25.19.155:53121->172.25.51.92:6783: write: connection reset by peer
INFO: 2018/09/05 03:42:57.520987 ->[172.25.51.92:6783|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: connection shutting down due to error: Multiple connections to d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 03:42:57.521137 ipsec: InitSALocal: 172.25.51.245 -> 172.25.19.155 :6784 0xdfb94507
INFO: 2018/09/05 03:42:57.522877 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)
INFO: 2018/09/05 03:42:57.522916 overlay_switch ->[d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)] fastdp send InitSARemote: write tcp4 172.25.19.155:53121->172.25.51.92:6783: write: connection reset by peer
INFO: 2018/09/05 03:42:57.533877 ipsec: InitSALocal: 172.25.19.192 -> 172.25.19.155 :6784 0xf020f3f4
INFO: 2018/09/05 03:42:57.544393 ipsec: InitSALocal: 172.25.83.167 -> 172.25.19.155 :6784 0x4cb07409
INFO: 2018/09/05 03:42:57.548738 ipsec: InitSALocal: 172.25.51.190 -> 172.25.19.155 :6784 0x583238f
ERRO: 2018/09/05 03:42:57.553527 fastdp ->[172.25.51.190:6784|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: ipsec init SA local failed: send InitSARemote: write tcp4 172.25.19.155:49401->172.25.51.190:6783: write: connection reset by peer
INFO: 2018/09/05 03:42:57.553560 ->[172.25.51.190:6783|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection shutting down due to error: Multiple connections to b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 03:42:57.553687 ipsec: InitSALocal: 172.25.19.165 -> 172.25.19.155 :6784 0x31ff5b8c
INFO: 2018/09/05 03:42:57.553945 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:42:57.553977 overlay_switch ->[b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)] fastdp send InitSARemote: write tcp4 172.25.19.155:49401->172.25.51.190:6783: write: connection reset by peer
INFO: 2018/09/05 03:42:57.561710 ipsec: InitSALocal: 172.25.51.92 -> 172.25.19.155 :6784 0x7b9f635f
INFO: 2018/09/05 03:42:57.571360 ipsec: InitSALocal: 172.25.51.190 -> 172.25.19.155 :6784 0x391bfee8
INFO: 2018/09/05 03:42:57.583259 ipsec: destroy: in 172.25.19.165 -> 172.25.19.155 0x5038c53c
INFO: 2018/09/05 03:42:57.594704 ipsec: destroy: in 172.25.51.92 -> 172.25.19.155 0x8cdc6e29
INFO: 2018/09/05 03:42:57.602649 ipsec: destroy: in 172.25.51.190 -> 172.25.19.155 0x583238f
INFO: 2018/09/05 03:42:57.756351 fastdp ->[172.25.83.103:6784|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.756380 ipsec: InitSARemote: 172.25.19.155 -> 172.25.83.103 :6784 0xa3186149
INFO: 2018/09/05 03:42:57.756855 fastdp ->[172.25.51.190:6784|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.756872 ipsec: InitSARemote: 172.25.19.155 -> 172.25.51.190 :6784 0x4b8ae545
INFO: 2018/09/05 03:42:57.756989 fastdp ->[172.25.83.156:6784|aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.757092 ipsec: InitSARemote: 172.25.19.155 -> 172.25.83.156 :6784 0x8de4c3df
INFO: 2018/09/05 03:42:57.757762 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.757881 overlay_switch ->[aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)] using sleeve
INFO: 2018/09/05 03:42:57.757900 ->[172.25.83.103:6783|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.758501 fastdp ->[172.25.19.165:6784|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.758514 ipsec: InitSARemote: 172.25.19.155 -> 172.25.19.165 :6784 0x19339967
INFO: 2018/09/05 03:42:57.758542 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.758619 overlay_switch ->[aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)] using sleeve
INFO: 2018/09/05 03:42:57.758644 ->[172.25.83.156:6783|aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.758857 fastdp ->[172.25.51.245:6784|d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.758870 ipsec: InitSARemote: 172.25.19.155 -> 172.25.51.245 :6784 0x11112a70
INFO: 2018/09/05 03:42:57.759508 fastdp ->[172.25.19.192:6784|ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.759521 ipsec: InitSARemote: 172.25.19.155 -> 172.25.19.192 :6784 0xd4e2e871
INFO: 2018/09/05 03:42:57.759595 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.759689 overlay_switch ->[d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)] using sleeve
INFO: 2018/09/05 03:42:57.759710 ->[172.25.51.245:6783|d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.760161 sleeve ->[172.25.83.103:6783|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:57.760176 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.760219 sleeve ->[172.25.83.156:6783|aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:57.760251 overlay_switch ->[ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)] using sleeve
INFO: 2018/09/05 03:42:57.760267 ->[172.25.19.192:6783|ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.760614 fastdp ->[172.25.51.92:6784|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.760629 ipsec: InitSARemote: 172.25.19.155 -> 172.25.51.92 :6784 0x869c15e9
INFO: 2018/09/05 03:42:57.761682 sleeve ->[172.25.51.245:6783|d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:57.761889 sleeve ->[172.25.19.192:6783|ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:57.762050 fastdp ->[172.25.83.167:6784|ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 03:42:57.762061 ipsec: InitSARemote: 172.25.19.155 -> 172.25.83.167 :6784 0x5bc1962f
INFO: 2018/09/05 03:42:57.762911 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.763006 overlay_switch ->[ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)] using sleeve
INFO: 2018/09/05 03:42:57.763030 ->[172.25.83.167:6783|ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.764983 overlay_switch ->[aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.765023 ->[172.25.19.165:42024|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.765168 overlay_switch ->[aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.765219 sleeve ->[172.25.83.167:6783|ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:57.765574 overlay_switch ->[ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.765602 overlay_switch ->[d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.765631 ->[172.25.51.92:60214|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.766060 overlay_switch ->[ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)] using fastdp
INFO: 2018/09/05 03:42:57.768501 ->[172.25.51.190:43183|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection fully established
INFO: 2018/09/05 03:42:57.787496 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.787761 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.790048 sleeve ->[172.25.19.165:6783|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:57.790153 sleeve ->[172.25.51.92:6783|d2:e6:2b:b1:f5:cf(ip-172-25-51-92.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:57.790671 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 03:42:57.792422 sleeve ->[172.25.51.190:6783|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 03:42:58.420098 [kube-peers] Added myself to peer list &{[{8e:bb:55:a5:8e:20 ip-172-25-19-10.ec2.internal} {aa:fa:7b:50:7e:1f ip-172-25-83-103.ec2.internal} {22:8a:ae:47:09:4d ip-172-25-19-165.ec2.internal} {b6:c4:03:b6:d7:52 ip-172-25-51-190.ec2.internal} {d6:95:93:7e:5e:1c ip-172-25-51-245.ec2.internal} {aa:d1:56:c1:fb:9f ip-172-25-83-156.ec2.internal} {d2:e6:2b:b1:f5:cf ip-172-25-51-92.ec2.internal} {ae:2a:74:a7:75:f4 ip-172-25-19-192.ec2.internal} {ea:a8:05:08:9f:aa ip-172-25-83-167.ec2.internal} {62:b1:3d:d5:2e:5e ip-172-25-19-155.ec2.internal}]}
DEBU: 2018/09/05 03:42:58.424040 [kube-peers] Nodes that have disappeared: map[ip-172-25-19-10.ec2.internal:{8e:bb:55:a5:8e:20 ip-172-25-19-10.ec2.internal}]
DEBU: 2018/09/05 03:42:58.424064 [kube-peers] Preparing to remove disappeared peer {8e:bb:55:a5:8e:20 ip-172-25-19-10.ec2.internal}
DEBU: 2018/09/05 03:42:58.424076 [kube-peers] Noting I plan to remove  8e:bb:55:a5:8e:20
DEBU: 2018/09/05 03:42:58.435996 weave DELETE to http://127.0.0.1:6784/peer/8e:bb:55:a5:8e:20 with map[]
INFO: 2018/09/05 03:42:58.452983 [kube-peers] rmpeer of 8e:bb:55:a5:8e:20: 614923 IPs taken over from 8e:bb:55:a5:8e:20

DEBU: 2018/09/05 03:42:58.488302 [kube-peers] Nodes that have disappeared: map[]
100.96.0.1
INFO: 2018/09/05 03:43:07.193904 Discovered remote MAC 02:bb:e6:dc:4a:fa at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:07.258794 Discovered remote MAC 8a:59:d8:b7:c0:8d at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:43:07.344980 Discovered remote MAC 82:86:fa:81:2f:3c at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:43:07.404905 Discovered remote MAC 22:a5:68:91:5e:58 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:43:07.474916 Discovered remote MAC 7e:66:ba:d0:0e:b9 at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:43:07.534957 Discovered remote MAC a6:6a:6c:22:b1:1e at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:43:07.608667 Discovered remote MAC 12:5b:4c:be:51:19 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:43:07.677333 Discovered remote MAC 36:24:36:f7:57:16 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:07.758975 Discovered remote MAC fe:b6:20:29:1a:58 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:07.819906 Discovered remote MAC 6a:36:9d:f0:cc:0f at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:43:07.892894 Discovered remote MAC 4e:1c:fe:43:7e:41 at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:43:07.960874 Discovered remote MAC de:4f:5b:9a:7e:5f at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:43:08.034697 Discovered remote MAC de:c8:81:27:35:0b at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:43:08.103909 Discovered remote MAC ee:01:57:88:74:da at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:43:08.547356 Discovered remote MAC f2:57:94:85:47:a3 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:43:09.168528 Discovered remote MAC 7e:e1:24:26:83:fd at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:43:09.235979 Discovered remote MAC 6a:01:27:74:fc:79 at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:43:09.366115 Discovered remote MAC 86:8d:7d:71:6b:94 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:43:09.439640 Discovered remote MAC 3e:93:cf:2e:4e:2f at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:43:09.506874 Discovered remote MAC fa:45:29:b4:c3:17 at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:43:09.575911 Discovered remote MAC 92:4c:56:09:d0:94 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:09.649577 Discovered remote MAC 8a:15:ed:ed:69:ac at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:09.714900 Discovered remote MAC 92:c8:1e:db:42:35 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:09.781938 Discovered remote MAC ea:46:cf:60:f3:85 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:43:09.847828 Discovered remote MAC ba:73:ff:30:9a:9a at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:43:09.907871 Discovered remote MAC ea:7b:df:fd:04:39 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:09.974943 Discovered remote MAC 0e:61:47:d8:c0:7e at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:43:10.044913 Discovered remote MAC 8a:e6:24:ab:aa:52 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:43:23.147494 Discovered remote MAC ea:a8:05:08:9f:aa at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:43:32.568903 Discovered remote MAC 76:72:89:4a:34:fe at aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)
INFO: 2018/09/05 03:43:32.647904 Discovered remote MAC ca:a8:55:58:ae:06 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:32.668969 Discovered remote MAC 8e:30:b7:7b:a6:d3 at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:43:48.743944 Discovered remote MAC 22:8a:ae:47:09:4d at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:43:57.409273 Discovered remote MAC aa:fa:7b:50:7e:1f at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:44:00.845063 Discovered remote MAC b6:c4:03:b6:d7:52 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:44:00.875045 Discovered remote MAC d6:95:93:7e:5e:1c at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:45:43.483235 Discovered remote MAC 62:68:a3:6d:f8:3e at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:46:08.628926 Discovered remote MAC 22:1b:07:5d:55:7a at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:49:23.928685 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 03:49:26.936702 Discovered remote MAC 46:3f:92:cd:32:c8 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:50:56.262370 Discovered remote MAC c6:1a:24:88:df:85 at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:51:06.082383 Discovered remote MAC 86:b6:4b:0a:bc:2f at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:51:06.294561 Discovered remote MAC 0e:fd:46:50:f2:1d at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:06.613527 Discovered remote MAC ea:6e:02:65:cc:6a at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:06.971698 Discovered remote MAC 12:ca:8b:20:95:22 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:07.870620 Discovered remote MAC 32:73:31:19:30:a4 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:08.331846 Discovered remote MAC d6:7f:ce:c7:2d:e1 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:08.622301 Discovered remote MAC d6:ef:24:7a:fc:ad at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:08.947137 Discovered remote MAC 26:a0:4e:f9:53:34 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:09.317199 Discovered remote MAC c6:a7:8d:6e:cf:b7 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:09.698010 Discovered remote MAC 2e:40:0b:08:1d:28 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:10.241312 Discovered remote MAC 62:90:25:90:e8:99 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 03:51:16.492732 Discovered remote MAC ba:6f:59:3c:5b:74 at aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 03:51:33.288334 Discovered remote MAC 06:df:b5:31:57:46 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 03:51:46.172191 Discovered remote MAC 4a:c4:31:c3:73:da at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:52:15.646637 Discovered remote MAC d6:33:1d:bb:ce:53 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:52:15.801755 Discovered remote MAC fe:d8:a5:b1:aa:84 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 03:52:15.894679 Discovered remote MAC 62:8c:bf:35:fb:e3 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:06:08.779603 ->[172.25.83.103:6783|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: connection shutting down due to error: read tcp4 172.25.19.155:48871->172.25.83.103:6783: read: connection reset by peer
INFO: 2018/09/05 04:06:08.779665 ->[172.25.83.103:6783|aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)]: connection deleted
INFO: 2018/09/05 04:06:08.782652 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:06:08.782936 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 04:06:08.782997 ipsec: destroy: in 172.25.83.103 -> 172.25.19.155 0x96a99a41
INFO: 2018/09/05 04:06:08.783771 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection refused
INFO: 2018/09/05 04:06:08.786090 Removed unreachable peer aa:fa:7b:50:7e:1f(ip-172-25-83-103.ec2.internal)
INFO: 2018/09/05 04:06:08.795585 ipsec: destroy: out 172.25.19.155 -> 172.25.83.103 0xa3186149
ERRO: 2018/09/05 04:06:08.857150 Captured frame from MAC (ea:a8:05:08:9f:aa) to (de:c8:81:27:35:0b) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:06:08.874233 Captured frame from MAC (8a:15:ed:ed:69:ac) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:06:08.924228 Captured frame from MAC (92:c8:1e:db:42:35) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:06:08.925854 Captured frame from MAC (22:8a:ae:47:09:4d) to (de:c8:81:27:35:0b) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:06:08.984663 Captured frame from MAC (ea:46:cf:60:f3:85) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:06:09.017771 Captured frame from MAC (d6:95:93:7e:5e:1c) to (de:c8:81:27:35:0b) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:06:09.051309 Captured frame from MAC (b6:c4:03:b6:d7:52) to (de:c8:81:27:35:0b) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:06:09.052171 Captured frame from MAC (92:4c:56:09:d0:94) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:06:09.103011 Captured frame from MAC (ba:73:ff:30:9a:9a) to (aa:fa:7b:50:7e:1f) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:06:09.129994 Captured frame from MAC (8a:e6:24:ab:aa:52) to (aa:fa:7b:50:7e:1f) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:06:09.149542 Captured frame from MAC (d6:7f:ce:c7:2d:e1) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:06:09.349538 Captured frame from MAC (0e:fd:46:50:f2:1d) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:06:09.351463 Captured frame from MAC (26:a0:4e:f9:53:34) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:06:09.391200 Captured frame from MAC (02:bb:e6:dc:4a:fa) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:06:09.411403 Captured frame from MAC (ea:7b:df:fd:04:39) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:06:09.549897 Captured frame from MAC (a6:6a:6c:22:b1:1e) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:06:09.710125 Captured frame from MAC (22:a5:68:91:5e:58) to (aa:fa:7b:50:7e:1f) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:06:09.730020 Captured frame from MAC (fe:b6:20:29:1a:58) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:06:09.905718 Captured frame from MAC (6a:36:9d:f0:cc:0f) to (aa:fa:7b:50:7e:1f) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:06:10.711592 Captured frame from MAC (c6:a7:8d:6e:cf:b7) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 04:06:10.719023 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:06:10.722915 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection refused
ERRO: 2018/09/05 04:06:10.740438 Captured frame from MAC (fa:45:29:b4:c3:17) to (aa:fa:7b:50:7e:1f) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 04:06:14.171164 ->[172.25.83.103:6783] attempting connection
ERRO: 2018/09/05 04:07:56.974586 Captured frame from MAC (ea:46:cf:60:f3:85) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:07:56.974729 Captured frame from MAC (8a:e6:24:ab:aa:52) to (aa:fa:7b:50:7e:1f) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:07:56.974896 Captured frame from MAC (d6:7f:ce:c7:2d:e1) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:07:56.975038 Captured frame from MAC (ba:73:ff:30:9a:9a) to (aa:fa:7b:50:7e:1f) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:07:56.975069 Captured frame from MAC (c6:a7:8d:6e:cf:b7) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:07:56.975095 Captured frame from MAC (92:4c:56:09:d0:94) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:07:56.975155 Captured frame from MAC (ea:a8:05:08:9f:aa) to (de:c8:81:27:35:0b) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:07:56.975181 Captured frame from MAC (92:c8:1e:db:42:35) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:07:56.975223 Captured frame from MAC (0e:fd:46:50:f2:1d) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:07:56.975251 Captured frame from MAC (b6:c4:03:b6:d7:52) to (de:c8:81:27:35:0b) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:07:56.975277 Captured frame from MAC (ea:7b:df:fd:04:39) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:07:56.975374 Captured frame from MAC (22:8a:ae:47:09:4d) to (de:c8:81:27:35:0b) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:07:56.975456 Captured frame from MAC (fa:45:29:b4:c3:17) to (aa:fa:7b:50:7e:1f) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:07:56.976094 Captured frame from MAC (26:a0:4e:f9:53:34) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:07:56.976151 Captured frame from MAC (d6:95:93:7e:5e:1c) to (de:c8:81:27:35:0b) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:07:56.976527 Captured frame from MAC (8a:15:ed:ed:69:ac) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
ERRO: 2018/09/05 04:07:56.976722 Captured frame from MAC (22:a5:68:91:5e:58) to (aa:fa:7b:50:7e:1f) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:07:56.976766 Captured frame from MAC (a6:6a:6c:22:b1:1e) to (aa:fa:7b:50:7e:1f) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:07:56.976806 Captured frame from MAC (02:bb:e6:dc:4a:fa) to (aa:fa:7b:50:7e:1f) associated with another peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 04:08:21.470982 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:08:24.817217 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:10:32.030972 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:10:38.679441 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:12:45.919012 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:12:55.314757 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:14:12.111255 ->[172.25.83.180:53523] connection accepted
INFO: 2018/09/05 04:14:12.116857 ->[172.25.83.180:53523|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 04:14:12.116940 overlay_switch ->[c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)] using fastdp
INFO: 2018/09/05 04:14:12.116970 ->[172.25.83.180:53523|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 04:14:12.118113 ->[172.25.83.180:6783] attempting connection
INFO: 2018/09/05 04:14:12.119547 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:12.120078 ipsec: InitSALocal: 172.25.83.180 -> 172.25.19.155 :6784 0x15acc079
INFO: 2018/09/05 04:14:12.144419 ->[172.25.83.180:6783|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 04:14:12.144501 overlay_switch ->[c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)] using fastdp
INFO: 2018/09/05 04:14:12.144521 ->[172.25.83.180:53523|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: connection deleted
INFO: 2018/09/05 04:14:12.144591 ->[172.25.83.180:6783|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 04:14:12.144652 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:12.144778 ipsec: InitSALocal: 172.25.83.180 -> 172.25.19.155 :6784 0x2bbbfce4
INFO: 2018/09/05 04:14:12.145166 ->[172.25.83.180:53523|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: connection shutting down due to error: Multiple connections to c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 04:14:12.145292 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:12.150511 ipsec: destroy: in 172.25.83.180 -> 172.25.19.155 0x15acc079
INFO: 2018/09/05 04:14:12.453347 sleeve ->[172.25.83.180:6783|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: Failed to decode packet: UDP packet decryption failed
INFO: 2018/09/05 04:14:12.510108 fastdp ->[172.25.83.180:6784|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 04:14:12.510139 ipsec: InitSARemote: 172.25.19.155 -> 172.25.83.180 :6784 0xb6fc3d2b
INFO: 2018/09/05 04:14:12.549613 Discovered remote MAC c6:53:80:65:ef:74 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:12.563364 Discovered remote MAC fe:c3:bb:a7:bc:d5 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:12.661103 overlay_switch ->[c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)] using sleeve
INFO: 2018/09/05 04:14:12.661135 ->[172.25.83.180:6783|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: connection fully established
INFO: 2018/09/05 04:14:12.661282 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 04:14:12.667974 sleeve ->[172.25.83.180:6783|c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 04:14:12.873297 Discovered remote MAC 72:ca:52:5a:f4:cd at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:13.012801 overlay_switch ->[c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)] using fastdp
INFO: 2018/09/05 04:14:20.766033 Discovered remote MAC f2:c6:0b:54:3b:b9 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:23.157332 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:23.179857 Discovered remote MAC 26:27:bb:c6:22:b6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:14:24.009479 Discovered remote MAC 4a:08:ba:ef:6b:ed at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:15:02.622985 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:15:23.793089 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:16:23.440204 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 04:16:31.838936 Discovered remote MAC 9e:a2:00:11:8b:bc at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:17:31.102913 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:17:57.354211 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:20:04.574945 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:20:25.210931 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:21:05.004168 Discovered remote MAC d6:ef:24:7a:fc:ad at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 04:21:05.560386 Discovered remote MAC a2:94:3d:b8:af:f6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:05.946084 Discovered remote MAC 52:0f:c5:1a:f4:e3 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:06.062466 Discovered remote MAC 46:3f:92:cd:32:c8 at 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 04:21:07.683361 Discovered remote MAC 6a:27:88:d0:d7:99 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:07.897098 Discovered remote MAC e6:28:40:a5:01:4c at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:08.240322 Discovered remote MAC 0a:9b:63:ad:d9:4d at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:08.431338 Discovered remote MAC 82:c0:57:43:fa:6f at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:09.405421 Discovered remote MAC 06:fe:e3:ed:b7:63 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:10.265668 Discovered remote MAC 8a:ae:04:f5:4c:88 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:10.858436 Discovered remote MAC aa:b1:73:da:2f:97 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:21:11.265711 Discovered remote MAC aa:e5:ce:d6:80:97 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:22:18.407805 Discovered remote MAC 06:df:b5:31:57:46 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:22:32.414942 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:23:42.910239 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:25:50.174945 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:26:53.246823 Discovered remote MAC ba:5c:31:60:bc:cf at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:27:25.969280 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:29:33.278995 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:31:31.280034 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:33:38.526952 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:36:09.628787 ->[172.25.19.165:42024|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection shutting down due to error: read tcp4 172.25.19.155:6783->172.25.19.165:42024: read: connection reset by peer
INFO: 2018/09/05 04:36:09.628869 ->[172.25.19.165:42024|22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)]: connection deleted
INFO: 2018/09/05 04:36:09.632057 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:36:09.632379 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 04:36:09.632414 ipsec: destroy: in 172.25.19.165 -> 172.25.19.155 0x31ff5b8c
INFO: 2018/09/05 04:36:09.632760 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: connection refused
INFO: 2018/09/05 04:36:09.634899 Removed unreachable peer 22:8a:ae:47:09:4d(ip-172-25-19-165.ec2.internal)
INFO: 2018/09/05 04:36:09.640121 ipsec: destroy: out 172.25.19.155 -> 172.25.19.165 0x19339967
ERRO: 2018/09/05 04:36:09.692676 Captured frame from MAC (7e:e1:24:26:83:fd) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:09.733124 Captured frame from MAC (22:a5:68:91:5e:58) to (22:8a:ae:47:09:4d) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:36:09.869033 Captured frame from MAC (12:5b:4c:be:51:19) to (22:8a:ae:47:09:4d) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:36:09.916872 Captured frame from MAC (86:8d:7d:71:6b:94) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:09.964787 Captured frame from MAC (c6:a7:8d:6e:cf:b7) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:09.969881 Captured frame from MAC (06:fe:e3:ed:b7:63) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:10.030246 Captured frame from MAC (ea:a8:05:08:9f:aa) to (fe:b6:20:29:1a:58) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:10.060630 Captured frame from MAC (8a:ae:04:f5:4c:88) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:10.080048 Captured frame from MAC (c6:53:80:65:ef:74) to (fe:b6:20:29:1a:58) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:10.104001 Captured frame from MAC (ea:46:cf:60:f3:85) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:10.160417 Captured frame from MAC (6a:01:27:74:fc:79) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:10.232537 Captured frame from MAC (fa:45:29:b4:c3:17) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:10.344576 Captured frame from MAC (aa:e5:ce:d6:80:97) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:10.364019 Captured frame from MAC (d6:95:93:7e:5e:1c) to (fe:b6:20:29:1a:58) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:10.460425 Captured frame from MAC (4e:1c:fe:43:7e:41) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:10.474605 Captured frame from MAC (82:c0:57:43:fa:6f) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:10.503175 Captured frame from MAC (0e:fd:46:50:f2:1d) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:10.607027 Captured frame from MAC (d6:7f:ce:c7:2d:e1) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:10.734925 Captured frame from MAC (b6:c4:03:b6:d7:52) to (fe:b6:20:29:1a:58) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:36:10.908711 Captured frame from MAC (a6:6a:6c:22:b1:1e) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:11.626536 Captured frame from MAC (6a:36:9d:f0:cc:0f) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:11.632841 Captured frame from MAC (62:90:25:90:e8:99) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 04:36:11.668765 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:36:11.669501 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: connection refused
INFO: 2018/09/05 04:36:14.448870 ->[172.25.19.165:6783] attempting connection
ERRO: 2018/09/05 04:36:14.664539 Captured frame from MAC (ea:a8:05:08:9f:aa) to (ca:a8:55:58:ae:06) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:36:17.403033 Captured frame from MAC (d6:95:93:7e:5e:1c) to (ca:a8:55:58:ae:06) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:23.039067 Captured frame from MAC (b6:c4:03:b6:d7:52) to (ca:a8:55:58:ae:06) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:36:28.042281 Captured frame from MAC (22:a5:68:91:5e:58) to (ca:a8:55:58:ae:06) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:36:31.637435 Captured frame from MAC (c6:53:80:65:ef:74) to (ca:a8:55:58:ae:06) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:31.637541 Captured frame from MAC (c6:53:80:65:ef:74) to (ca:a8:55:58:ae:06) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:33.927752 Captured frame from MAC (fa:45:29:b4:c3:17) to (ca:a8:55:58:ae:06) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:36:41.412770 Captured frame from MAC (06:fe:e3:ed:b7:63) to (ca:a8:55:58:ae:06) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:36:41.642468 Captured frame from MAC (6a:27:88:d0:d7:99) to (ca:a8:55:58:ae:06) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:36:48.508975 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:36:53.089660 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:36:56.095009 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:37:04.020256 Discovered remote MAC a2:07:06:e4:c6:07 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:37:04.199011 Discovered remote MAC ea:1d:a0:1c:d9:48 at ae:2a:74:a7:75:f4(ip-172-25-19-192.ec2.internal)
INFO: 2018/09/05 04:37:04.877823 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:37:07.882969 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:37:12.533117 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:37:22.846755 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:37:25.853026 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:37:37.374426 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:37:40.379018 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
ERRO: 2018/09/05 04:37:56.974883 Captured frame from MAC (d6:95:93:7e:5e:1c) to (ca:a8:55:58:ae:06) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:37:56.974939 Captured frame from MAC (d6:95:93:7e:5e:1c) to (fe:b6:20:29:1a:58) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:37:56.974974 Captured frame from MAC (ea:a8:05:08:9f:aa) to (ca:a8:55:58:ae:06) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.975164 Captured frame from MAC (7e:e1:24:26:83:fd) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:37:56.975213 Captured frame from MAC (c6:53:80:65:ef:74) to (ca:a8:55:58:ae:06) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:37:56.975404 Captured frame from MAC (ea:a8:05:08:9f:aa) to (fe:b6:20:29:1a:58) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.975431 Captured frame from MAC (d6:7f:ce:c7:2d:e1) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.975456 Captured frame from MAC (6a:36:9d:f0:cc:0f) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:37:56.975514 Captured frame from MAC (86:8d:7d:71:6b:94) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.975699 Captured frame from MAC (fa:45:29:b4:c3:17) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:37:56.975785 Captured frame from MAC (82:c0:57:43:fa:6f) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:37:56.975832 Captured frame from MAC (22:a5:68:91:5e:58) to (22:8a:ae:47:09:4d) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:37:56.975884 Captured frame from MAC (c6:53:80:65:ef:74) to (fe:b6:20:29:1a:58) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:37:56.975917 Captured frame from MAC (62:90:25:90:e8:99) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.976120 Captured frame from MAC (c6:a7:8d:6e:cf:b7) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.976403 Captured frame from MAC (12:5b:4c:be:51:19) to (22:8a:ae:47:09:4d) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:37:56.976933 Captured frame from MAC (8a:ae:04:f5:4c:88) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:37:56.977206 Captured frame from MAC (06:fe:e3:ed:b7:63) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:37:56.978030 Captured frame from MAC (0e:fd:46:50:f2:1d) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.978216 Captured frame from MAC (ea:46:cf:60:f3:85) to (22:8a:ae:47:09:4d) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 04:37:56.978331 Captured frame from MAC (aa:e5:ce:d6:80:97) to (22:8a:ae:47:09:4d) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 04:37:56.978365 Captured frame from MAC (b6:c4:03:b6:d7:52) to (fe:b6:20:29:1a:58) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
ERRO: 2018/09/05 04:37:56.978523 Captured frame from MAC (6a:01:27:74:fc:79) to (22:8a:ae:47:09:4d) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 04:37:56.978613 Captured frame from MAC (b6:c4:03:b6:d7:52) to (ca:a8:55:58:ae:06) associated with another peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:38:06.444128 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:38:09.451951 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:39:15.333609 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:39:18.338987 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:39:19.775916 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:40:05.716902 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:40:08.722997 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:41:33.304768 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:41:36.310974 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:42:47.971674 ->[172.25.19.211:53072] connection accepted
INFO: 2018/09/05 04:42:47.973655 ->[172.25.19.211:53072|46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 04:42:47.973736 overlay_switch ->[46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)] using fastdp
INFO: 2018/09/05 04:42:47.973765 ->[172.25.19.211:53072|46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 04:42:47.977211 ->[172.25.19.211:6783] attempting connection
INFO: 2018/09/05 04:42:47.977875 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:42:47.977968 ipsec: InitSALocal: 172.25.19.211 -> 172.25.19.155 :6784 0x1b2420a2
INFO: 2018/09/05 04:42:48.015912 ->[172.25.19.211:6783|46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 04:42:48.016002 overlay_switch ->[46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)] using fastdp
INFO: 2018/09/05 04:42:48.016029 ->[172.25.19.211:6783|46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)]: connection shutting down due to error: Multiple connections to 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 04:42:48.410642 fastdp ->[172.25.19.211:6784|46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 04:42:48.410674 ipsec: InitSARemote: 172.25.19.155 -> 172.25.19.211 :6784 0x75ef24d2
INFO: 2018/09/05 04:42:48.414844 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 04:42:48.414942 overlay_switch ->[46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)] using sleeve
INFO: 2018/09/05 04:42:48.414962 ->[172.25.19.211:53072|46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)]: connection fully established
INFO: 2018/09/05 04:42:48.415898 sleeve ->[172.25.19.211:6783|46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 04:42:48.579655 Discovered remote MAC 9e:d4:ab:79:22:d4 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:42:48.586422 Discovered remote MAC be:c8:04:b7:82:79 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:42:48.590361 Discovered remote MAC 46:58:cc:8c:bc:7a at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:42:48.913369 overlay_switch ->[46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)] using fastdp
INFO: 2018/09/05 04:42:57.491984 Discovered remote MAC f6:6f:1a:29:ae:c6 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:42:59.612264 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:42:59.782919 Discovered remote MAC 22:b9:83:52:88:55 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:43:00.358364 Discovered remote MAC 9a:4c:b6:18:ef:79 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 04:43:19.424715 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:43:59.140541 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:44:02.147023 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:45:26.622942 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:49:15.727648 Discovered remote MAC 22:1b:07:5d:55:7a at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:49:24.930236 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 04:49:40.840745 Discovered remote MAC aa:d1:56:c1:fb:9f at aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)
INFO: 2018/09/05 04:50:02.178731 ->[172.25.51.190:43183|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection shutting down due to error: read tcp4 172.25.19.155:6783->172.25.51.190:43183: i/o timeout
INFO: 2018/09/05 04:50:02.178827 ->[172.25.51.190:43183|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection deleted
INFO: 2018/09/05 04:50:02.181230 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 04:50:02.181554 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:50:02.181582 ipsec: destroy: in 172.25.51.190 -> 172.25.19.155 0x391bfee8
INFO: 2018/09/05 04:50:02.189162 ipsec: destroy: out 172.25.19.155 -> 172.25.51.190 0x4b8ae545
INFO: 2018/09/05 04:50:09.438407 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:50:12.442976 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 04:50:15.198804 Removed unreachable peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:50:27.471835 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 04:52:09.438927 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 04:52:13.729958 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 04:52:13.731185 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection refused
INFO: 2018/09/05 04:52:17.675227 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 04:52:17.676583 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection refused
INFO: 2018/09/05 04:52:18.595212 ->[172.25.51.190:39723] connection accepted
INFO: 2018/09/05 04:52:18.608965 ->[172.25.51.190:39723|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 04:52:18.609054 overlay_switch ->[b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)] using fastdp
INFO: 2018/09/05 04:52:18.609085 ->[172.25.51.190:39723|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 04:52:18.609174 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:18.609695 ipsec: InitSALocal: 172.25.51.190 -> 172.25.19.155 :6784 0x8e2d173
INFO: 2018/09/05 04:52:19.172749 Discovered remote MAC b6:c4:03:b6:d7:52 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.278122 Discovered remote MAC a6:65:33:8c:e8:b7 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.291732 Discovered remote MAC 46:39:7e:f9:6a:b7 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.291911 Discovered remote MAC 22:95:b9:c6:d2:34 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.432876 Discovered remote MAC 36:0e:7c:d4:98:e2 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.433008 Discovered remote MAC 12:9a:73:04:d3:12 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.474305 Discovered remote MAC c6:e9:b5:83:44:d8 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.768857 Discovered remote MAC 0a:e4:56:84:56:73 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:19.981069 Discovered remote MAC c6:b3:09:33:dd:1f at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:20.103878 fastdp ->[172.25.51.190:6784|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 04:52:20.103912 ipsec: InitSARemote: 172.25.19.155 -> 172.25.51.190 :6784 0x7765de68
INFO: 2018/09/05 04:52:20.113556 overlay_switch ->[b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)] using sleeve
INFO: 2018/09/05 04:52:20.113588 ->[172.25.51.190:39723|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection fully established
INFO: 2018/09/05 04:52:20.113685 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 04:52:20.136724 sleeve ->[172.25.51.190:6783|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 04:52:20.368888 Discovered remote MAC ea:e9:95:4f:f5:71 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:20.373680 Discovered remote MAC b6:ea:cd:a8:92:a5 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:20.529806 Discovered remote MAC fa:25:26:00:18:70 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:20.595645 Discovered remote MAC e6:27:62:c2:78:70 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:20.606322 overlay_switch ->[b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)] using fastdp
INFO: 2018/09/05 04:52:20.736751 Discovered remote MAC 92:b1:ca:69:e4:fa at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:20.857133 Discovered remote MAC 86:ed:7d:08:16:d1 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:20.964287 Discovered remote MAC ba:3f:77:e4:5a:33 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:21.699532 Discovered remote MAC b2:2d:81:98:1c:b8 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:22.340474 Discovered remote MAC 7e:2b:93:27:f3:24 at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:23.112369 Discovered remote MAC b6:f4:9e:a1:07:1e at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:24.168233 Discovered remote MAC b6:c6:5e:3a:7d:3c at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:52:26.346695 Discovered remote MAC ae:1e:7b:b8:fe:cf at b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 04:53:03.038178 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 04:53:08.962228 Discovered remote MAC 6a:27:88:d0:d7:99 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 04:55:10.302936 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 04:57:50.812340 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 04:57:53.818974 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:00:53.164815 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 05:02:17.939433 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 05:03:00.446951 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 05:05:29.206314 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:05:32.211015 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:06:06.141622 Discovered remote MAC 52:90:98:b1:0f:4d at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:07.439389 Discovered remote MAC 8e:e4:30:a7:22:11 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:07.934275 Discovered remote MAC ce:de:04:92:76:f2 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:08.738854 Discovered remote MAC 52:92:d0:2f:c8:71 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:10.527360 Discovered remote MAC 0e:57:76:42:2d:35 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:10.967193 Discovered remote MAC 3a:b2:77:60:f4:97 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:11.071620 Discovered remote MAC ae:de:f7:68:8a:2d at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:11.955359 Discovered remote MAC ce:36:fa:4e:f7:84 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:12.238327 Discovered remote MAC 66:0f:de:2f:df:f6 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:12.479915 Discovered remote MAC be:38:d1:8e:6c:6d at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:12.745407 Discovered remote MAC 02:70:ea:9f:69:3f at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:06:13.030396 Discovered remote MAC 2e:0a:81:b4:ce:81 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:07:48.302203 Discovered remote MAC f6:6f:1a:29:ae:c6 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:10:13.185993 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 05:12:20.446968 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 05:12:30.069594 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:12:33.074995 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:16:24.471174 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 05:20:10.240004 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 05:20:11.467176 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:20:14.473008 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:21:09.816691 ->[172.25.51.190:39723|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection shutting down due to error: read tcp4 172.25.19.155:6783->172.25.51.190:39723: read: connection reset by peer
INFO: 2018/09/05 05:21:09.816759 ->[172.25.51.190:39723|b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)]: connection deleted
INFO: 2018/09/05 05:21:09.819844 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:21:09.820395 Destroying IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 05:21:09.820433 ipsec: destroy: in 172.25.51.190 -> 172.25.19.155 0x8e2d173
INFO: 2018/09/05 05:21:09.821458 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection refused
INFO: 2018/09/05 05:21:09.823019 Removed unreachable peer b6:c4:03:b6:d7:52(ip-172-25-51-190.ec2.internal)
INFO: 2018/09/05 05:21:09.827733 ipsec: destroy: out 172.25.19.155 -> 172.25.51.190 0x7765de68
ERRO: 2018/09/05 05:21:09.837060 Captured frame from MAC (0e:57:76:42:2d:35) to (b6:c4:03:b6:d7:52) associated with another peer 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
ERRO: 2018/09/05 05:21:09.949226 Captured frame from MAC (8a:ae:04:f5:4c:88) to (b6:c4:03:b6:d7:52) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 05:21:09.949339 Captured frame from MAC (aa:e5:ce:d6:80:97) to (b6:c4:03:b6:d7:52) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 05:21:10.017871 Captured frame from MAC (c6:a7:8d:6e:cf:b7) to (b6:c4:03:b6:d7:52) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 05:21:10.108227 Captured frame from MAC (ce:de:04:92:76:f2) to (b6:c4:03:b6:d7:52) associated with another peer 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
ERRO: 2018/09/05 05:21:10.117031 Captured frame from MAC (46:58:cc:8c:bc:7a) to (e6:27:62:c2:78:70) associated with another peer 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
ERRO: 2018/09/05 05:21:10.133871 Captured frame from MAC (0e:fd:46:50:f2:1d) to (b6:c4:03:b6:d7:52) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 05:21:10.181199 Discovered remote MAC aa:d1:56:c1:fb:9f at aa:d1:56:c1:fb:9f(ip-172-25-83-156.ec2.internal)
ERRO: 2018/09/05 05:21:10.357238 Captured frame from MAC (3a:b2:77:60:f4:97) to (b6:c4:03:b6:d7:52) associated with another peer 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
ERRO: 2018/09/05 05:21:10.397099 Captured frame from MAC (82:c0:57:43:fa:6f) to (b6:c4:03:b6:d7:52) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 05:21:10.452991 Captured frame from MAC (d6:7f:ce:c7:2d:e1) to (b6:c4:03:b6:d7:52) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 05:21:10.459874 Captured frame from MAC (26:a0:4e:f9:53:34) to (b6:c4:03:b6:d7:52) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 05:21:10.508362 Captured frame from MAC (8e:e4:30:a7:22:11) to (b6:c4:03:b6:d7:52) associated with another peer 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
ERRO: 2018/09/05 05:21:10.525273 Captured frame from MAC (f2:c6:0b:54:3b:b9) to (b6:c4:03:b6:d7:52) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 05:21:10.592122 Captured frame from MAC (a2:94:3d:b8:af:f6) to (b6:c4:03:b6:d7:52) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 05:21:10.606944 Captured frame from MAC (a6:6a:6c:22:b1:1e) to (b6:c4:03:b6:d7:52) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 05:21:10.742971 Captured frame from MAC (ea:6e:02:65:cc:6a) to (b6:c4:03:b6:d7:52) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 05:21:10.851641 Captured frame from MAC (4e:1c:fe:43:7e:41) to (b6:c4:03:b6:d7:52) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 05:21:10.938672 Captured frame from MAC (6a:36:9d:f0:cc:0f) to (b6:c4:03:b6:d7:52) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 05:21:11.073089 Captured frame from MAC (9a:4c:b6:18:ef:79) to (b6:c4:03:b6:d7:52) associated with another peer 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
ERRO: 2018/09/05 05:21:11.259344 Captured frame from MAC (7e:e1:24:26:83:fd) to (b6:c4:03:b6:d7:52) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 05:21:11.309054 Captured frame from MAC (52:90:98:b1:0f:4d) to (b6:c4:03:b6:d7:52) associated with another peer 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
ERRO: 2018/09/05 05:21:11.396932 Captured frame from MAC (ea:a8:05:08:9f:aa) to (e6:27:62:c2:78:70) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 05:21:11.408433 Captured frame from MAC (d6:95:93:7e:5e:1c) to (e6:27:62:c2:78:70) associated with another peer d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
ERRO: 2018/09/05 05:21:11.546719 Captured frame from MAC (c6:53:80:65:ef:74) to (e6:27:62:c2:78:70) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
ERRO: 2018/09/05 05:21:11.642974 Captured frame from MAC (ea:46:cf:60:f3:85) to (b6:c4:03:b6:d7:52) associated with another peer ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
ERRO: 2018/09/05 05:21:11.753097 Captured frame from MAC (06:fe:e3:ed:b7:63) to (b6:c4:03:b6:d7:52) associated with another peer c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 05:21:24.707822 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:21:45.708622 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 05:22:05.191125 Discovered remote MAC d6:0b:7c:f4:01:8f at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 05:22:05.470112 Discovered remote MAC aa:35:af:82:e2:46 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:22:17.438941 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 05:22:25.324844 Discovered remote MAC 26:27:bb:c6:22:b6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 05:23:31.934953 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:23:41.928803 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:24:29.469737 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:24:32.474994 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:25:49.150943 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:26:11.957252 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:28:19.166976 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:28:43.607004 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:29:09.831208 ->[172.25.51.82:37023] connection accepted
INFO: 2018/09/05 05:29:09.834295 ->[172.25.51.82:6783] attempting connection
INFO: 2018/09/05 05:29:09.836957 ->[172.25.51.82:37023|6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 05:29:09.837065 overlay_switch ->[6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)] using fastdp
INFO: 2018/09/05 05:29:09.837088 ->[172.25.51.82:37023|6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)]: connection added (new peer)
INFO: 2018/09/05 05:29:09.837239 Setting up IPsec between 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal) and 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:29:09.837333 ipsec: InitSALocal: 172.25.51.82 -> 172.25.19.155 :6784 0x814a1a06
INFO: 2018/09/05 05:29:09.845658 ->[172.25.51.82:6783|6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/09/05 05:29:09.845733 overlay_switch ->[6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)] using fastdp
INFO: 2018/09/05 05:29:09.845761 ->[172.25.51.82:6783|6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)]: connection shutting down due to error: Multiple connections to 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal) added to 62:b1:3d:d5:2e:5e(ip-172-25-19-155.ec2.internal)
INFO: 2018/09/05 05:29:10.458569 fastdp ->[172.25.51.82:6784|6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)]: IPSec init SA remote
INFO: 2018/09/05 05:29:10.458599 ipsec: InitSARemote: 172.25.19.155 -> 172.25.51.82 :6784 0x1e96d094
INFO: 2018/09/05 05:29:10.460856 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/09/05 05:29:10.460954 overlay_switch ->[6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)] using sleeve
INFO: 2018/09/05 05:29:10.460974 ->[172.25.51.82:37023|6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)]: connection fully established
INFO: 2018/09/05 05:29:10.462627 sleeve ->[172.25.51.82:6783|6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)]: Effective MTU verified at 8915
INFO: 2018/09/05 05:29:10.569061 Discovered remote MAC 6e:fb:50:a9:9d:8e at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:29:10.754234 Discovered remote MAC da:f1:87:2a:ba:8a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:29:10.960912 overlay_switch ->[6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)] using fastdp
INFO: 2018/09/05 05:29:11.494440 Discovered remote MAC ca:39:84:dc:cd:bd at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:29:20.384083 Discovered remote MAC 56:29:49:c4:3f:a4 at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:29:20.432442 Discovered remote MAC ea:ba:57:60:fe:0f at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:29:20.494451 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:29:24.383378 Discovered remote MAC d2:91:3e:9b:83:07 at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 05:30:48.034388 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 05:30:50.846914 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:31:33.454588 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:32:04.461387 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:32:07.471951 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:32:55.262928 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 05:33:40.702993 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:35:09.360114 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:37:16.639046 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:38:38.101076 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:38:41.106944 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:39:38.417201 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:41:31.158795 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 05:41:45.694954 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:42:03.578171 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 05:43:36.783779 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 05:43:38.462923 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 05:44:04.400345 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:45:28.699777 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:45:31.707004 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:46:11.678931 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:49:25.941396 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 05:50:37.464494 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 05:51:12.069467 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 05:51:26.226787 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:51:29.232982 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:52:09.617636 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:53:19.326942 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 05:54:16.926986 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 05:54:50.606830 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:54:53.612977 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 05:58:37.554767 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 05:59:32.290727 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 05:59:35.297022 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:00:26.175063 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:00:44.766941 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 06:02:29.940828 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 06:02:33.438937 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 06:05:29.825345 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:05:32.831058 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:06:39.784539 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 06:07:04.902747 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 06:09:12.158922 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 06:09:48.533456 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:09:57.696020 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:10:00.702967 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:11:55.742951 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 06:14:57.712898 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 06:15:50.488117 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 06:16:25.465893 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 06:17:17.522714 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:17:57.726914 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 06:18:06.188932 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:18:09.195003 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:19:24.766936 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 06:23:12.576162 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 06:23:28.611798 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:25:35.838944 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 06:26:26.225898 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 06:26:52.167431 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:26:55.173001 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:28:33.503937 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 06:29:55.534813 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 06:30:47.134043 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 06:32:03.529378 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:33:18.960188 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 06:33:58.990495 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:34:01.995022 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:34:10.782966 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 06:35:26.174934 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 06:36:46.483282 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 06:38:20.131164 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:40:27.358930 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 06:40:28.184324 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 06:41:56.784549 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:41:59.790994 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:42:35.358947 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 06:43:37.787129 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 06:45:59.772970 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:46:02.779016 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:48:57.235976 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:49:26.940141 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 06:50:08.965473 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:50:11.971031 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:50:47.467394 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 06:51:02.356523 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 06:51:04.542939 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 06:52:12.575180 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 06:53:09.598935 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 06:54:34.866194 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 06:55:32.996431 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 06:55:36.003099 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 06:56:42.078980 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:01:26.133760 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:01:29.139086 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:02:00.772034 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:02:24.355265 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:03:07.939042 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 07:04:08.030936 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:04:31.582948 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:06:42.784561 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 07:06:57.242321 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:07:00.246977 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:08:23.639944 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:09:45.350669 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 07:10:30.878932 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:10:48.939960 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:10:51.946989 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:11:01.500890 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 07:11:43.098271 Discovered remote MAC 02:70:ea:9f:69:3f at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 07:12:48.184930 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:13:56.256313 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:13:59.262963 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:14:55.454950 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:16:05.521889 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:18:12.702921 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:18:29.806994 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:20:37.086947 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:21:22.033944 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:21:25.039440 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:22:06.382269 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:23:25.573919 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 07:24:13.663901 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:25:46.227823 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:27:53.566938 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:29:56.603876 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 07:30:03.340116 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:30:06.346956 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:30:33.435892 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:32:40.670939 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:32:48.004928 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:34:55.198946 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:37:55.753523 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:37:58.758981 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:38:45.564211 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:40:52.830985 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:41:25.209591 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:43:32.446943 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:43:38.780934 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 07:43:39.276383 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 07:44:32.481994 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 07:46:25.493355 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:46:35.965077 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:46:38.970985 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:47:03.432567 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:48:32.734946 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:49:10.750929 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 07:49:27.937336 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 07:50:57.468406 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 07:51:54.594555 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 07:54:01.822955 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 07:54:35.663889 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 07:54:38.671894 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 07:58:04.837993 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 07:59:01.537207 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:00:12.126938 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 08:01:08.830926 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:02:03.574546 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 08:02:13.952827 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:02:16.958996 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:04:50.505618 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:06:49.784550 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 08:06:57.759005 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:07:53.929583 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 08:09:15.523347 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:09:18.528989 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:10:01.182944 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 08:14:59.714197 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 08:15:40.970348 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:16:14.105559 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 08:16:27.468129 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 08:16:45.486516 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:16:48.490992 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:17:48.255020 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:18:21.278944 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 08:21:05.375021 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:23:12.606954 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:23:15.272376 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:23:18.279011 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:24:12.572002 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 08:24:34.286949 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 08:26:41.502955 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 08:27:31.278248 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:28:44.114937 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:28:47.120984 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:29:38.526998 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:29:57.592166 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 08:35:28.314951 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 08:36:15.952890 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:36:18.958970 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:37:35.583917 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 08:38:31.963579 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:40:39.134940 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:43:39.780070 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 08:43:40.277074 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 08:43:43.673171 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:44:56.632493 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:44:59.638987 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:45:27.214813 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 08:45:50.942932 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:47:34.495997 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 08:49:17.195810 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:49:20.203006 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:49:28.937559 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 08:51:07.467995 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 08:52:53.984265 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 08:52:56.481289 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 08:55:01.214936 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 08:55:10.281268 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 08:57:17.534913 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 08:57:21.270130 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 08:57:24.274982 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 08:59:06.471418 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 09:01:13.694918 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 09:02:27.161619 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 09:04:34.398944 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 09:06:15.747534 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:06:18.752983 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:06:52.779507 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 09:07:57.071748 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 09:09:59.156971 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:10:02.163968 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:10:04.254911 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 09:10:14.747425 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 09:11:52.063845 Discovered remote MAC 02:70:ea:9f:69:3f at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 09:12:21.982933 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 09:13:14.577289 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 09:15:27.362438 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:15:30.367066 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:16:28.470862 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 09:17:50.351223 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 09:18:39.674670 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 09:19:57.662994 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 09:20:46.942934 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 09:23:57.944252 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:24:00.951007 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:24:25.571041 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 09:24:28.857656 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 09:26:36.126970 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 09:27:44.202970 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 09:29:51.454940 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 09:29:58.596957 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 09:31:32.424980 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:31:35.430985 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:34:49.487011 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 09:35:34.450303 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 09:36:56.798935 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 09:37:41.726987 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 09:38:43.397163 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:38:46.403129 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:42:20.917094 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:42:23.923083 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:43:16.927115 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 09:43:40.780053 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 09:43:41.276245 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 09:45:24.190950 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 09:46:13.964898 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 09:46:40.076930 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:46:43.082985 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:48:21.215940 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 09:49:29.936976 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 09:49:52.498084 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 09:51:55.415884 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 09:53:13.262372 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 09:54:02.590908 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 09:54:45.350722 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 09:55:13.609425 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 09:55:16.614975 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 09:55:20.542941 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 10:00:51.188854 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:00:54.206983 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:02:34.490271 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:04:17.531937 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 10:04:41.822964 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:06:24.734954 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 10:06:28.583367 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:06:31.589007 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:06:59.783359 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 10:09:12.399622 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:11:19.646898 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:12:58.993106 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:13:01.999010 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:14:26.492551 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 10:15:01.715653 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 10:16:29.472606 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 10:16:33.758977 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 10:17:33.169659 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:18:24.458395 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:18:27.462960 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:19:40.382944 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:24:35.767978 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 10:25:12.571947 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 10:26:43.039016 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 10:27:01.981771 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:27:04.986962 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:27:10.080031 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:27:17.130953 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 10:29:17.406931 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:29:44.380902 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 10:29:59.593550 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 10:31:51.646903 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 10:33:23.332454 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:35:30.654931 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:35:33.916863 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:35:36.922993 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:39:18.850404 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:39:43.849970 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 10:41:26.114964 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:41:51.070946 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 10:43:41.779240 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 10:44:10.831337 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:44:13.837912 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:44:51.773020 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:46:59.038934 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:49:30.937054 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 10:50:12.981811 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 10:51:19.800889 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 10:51:27.475836 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 10:51:31.077460 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:51:34.083094 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:52:20.318947 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 10:53:27.134991 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 10:57:51.594938 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 10:57:54.600979 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 10:57:54.894032 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:00:02.142956 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:01:16.241931 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 11:02:53.736694 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:02:56.743007 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:03:23.487002 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 11:07:02.783589 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 11:07:41.522326 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:08:57.859291 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 11:09:30.994922 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:09:34.001003 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:09:48.766996 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:09:56.483179 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 11:10:48.502233 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 11:11:05.182974 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 11:12:01.026191 Discovered remote MAC 02:70:ea:9f:69:3f at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 11:14:13.982803 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:14:16.989134 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:17:11.619200 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:18:44.065230 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:18:47.070988 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:19:18.878949 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:19:48.358140 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 11:21:55.678950 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 11:22:23.334317 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:22:26.338980 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:25:25.570453 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 11:27:32.648779 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:29:09.480059 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 11:29:39.934963 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:30:00.592654 Discovered remote MAC d2:99:88:c3:44:9a at 6e:fb:50:a9:9d:8e(ip-172-25-51-82.ec2.internal)
INFO: 2018/09/05 11:30:44.528257 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:30:47.535026 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:31:16.702931 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 11:33:16.213643 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:34:47.573521 Discovered remote MAC c6:5b:37:18:66:a6 at c6:53:80:65:ef:74(ip-172-25-83-180.ec2.internal)
INFO: 2018/09/05 11:35:23.486906 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:36:54.142525 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 11:37:36.802767 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:37:39.808998 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:39:01.470949 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 11:40:43.359660 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:42:50.590946 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:43:42.779970 Discovered remote MAC e6:82:b3:3e:22:12 at 46:58:cc:8c:bc:7a(ip-172-25-19-211.ec2.internal)
INFO: 2018/09/05 11:45:20.835734 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:45:23.843018 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:45:46.131087 Discovered remote MAC 32:46:9c:36:92:ef at d6:95:93:7e:5e:1c(ip-172-25-51-245.ec2.internal)
INFO: 2018/09/05 11:46:02.044648 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 11:46:10.137921 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:48:09.310944 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
INFO: 2018/09/05 11:48:17.374951 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:51:37.469526 Discovered remote MAC 96:6f:37:8b:25:32 at ea:a8:05:08:9f:aa(ip-172-25-83-167.ec2.internal)
INFO: 2018/09/05 11:51:55.569162 ->[172.25.19.165:6783] attempting connection
INFO: 2018/09/05 11:51:58.575013 ->[172.25.19.165:6783] error during connection attempt: dial tcp4 :0->172.25.19.165:6783: connect: no route to host
INFO: 2018/09/05 11:52:38.835318 ->[172.25.83.103:6783] attempting connection
INFO: 2018/09/05 11:52:54.819510 ->[172.25.51.190:6783] attempting connection
INFO: 2018/09/05 11:54:46.111038 ->[172.25.83.103:6783] error during connection attempt: dial tcp4 :0->172.25.83.103:6783: connect: connection timed out
INFO: 2018/09/05 11:55:02.110961 ->[172.25.51.190:6783] error during connection attempt: dial tcp4 :0->172.25.51.190:6783: connect: connection timed out
@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 5, 2018

@bboreham

Is that what you meant to write? Wouldn't we expect a new peer to have both a new host name and a different MAC ?

Sorry, no i meant existing host name with new MAC. In our case, its pretty common to get a new AWS node with an ip that existed before, but that is in fact a new AWS node with a new MAC. I suspect this is what creates 'big problems'

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 5, 2018

@dcowden really, Weave Net does not note or care about host MACs.

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 5, 2018

@bboreham OH I SEE! yes, what i'm calling MAC is your peer id, because it looks like a MAC. Consider me corrected.

Meanwhile, would you like me to create a new issue and post all of my information on that new one? And is there anything else you need to be able to make a solution?

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 5, 2018

So, the design has nothing to do with IP addresses.

How is the peer ID calculated? if the answer is the host name (or something that is unique within a cluster only by host name), then in fact on AWS the ip address does matter, because the only difference between node host names is the ip address, because the host name is basically an ip address:

ip-172-25-19-155.ec2.internal

That said, I'm not sure it matters all that much. These arguments are only relevant because of guesses that i've made. Feel free to discard my comments and suppositions except for this one:

weave 2.4.0 does not clean up peers correctly, and it is repeatable, producable behavior when you terminate nodes within the same ASG

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 5, 2018

Yes that was my guess as well, that the fact that the IP addresses as part of the hostname matters. And again, one approach could be to find a way to implement some sort of timeout when the nodes are gone for too long... but I guess I need to read the source code to understand if this is even feasible.

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 5, 2018

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 5, 2018

Yes, please open a new issue for each separate case, this helps to keep the threads of conversation clear.

If in doubt as to whether it is a separate case, open a new issue.

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 5, 2018

@bboreham I have created this issue and re-posted relevant stuff for our issue

Please let me know if you need more information to make progress. From a technical viewpoint, I believe it is nearly certain that my new issue is in fact the exact same as this one ( which is why i commented on it). They both have the same exact root cause and scenario: AWS nodes terminating and then coming back as a part of an ASG.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 5, 2018

related: #3394

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 6, 2018

Sorry, I had missed #3394.

None of those are in the peer list, so they won't get cleaned up now.

However this one:

ba:57:58:4e:78:04(ip-10-11-52-39.us-west-1.compute.internal)    49152 IPs (02.3% of total)

has a different peer with the same hostname in the peer list:

INFO: 2018/09/05 08:02:50.003952 [kube-peers] Added myself to peer list &{[
{9a:6b:df:34:bd:5a ip-10-11-46-213.us-west-1.compute.internal} 
{ba:57:58:4e:78:77 ip-10-11-52-39.us-west-1.compute.internal}
[...]

which triggers the problem described there.

@dcowden

This comment has been minimized.

Copy link

dcowden commented Sep 6, 2018

Ok so basically on aws IP address does matter because it's part of the host name.

To me it seems clear that when we see a new peer with the same host name, we should remove the old one.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 6, 2018

@dcowden and/or probably also use something more unique than the hostname like the Kubernetes Node ID (which AFAIK should be unique) as suggested in #3394

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 6, 2018

Note we still haven't explained how the unreachable peer IDs (e.g. ba:57:58:4e:78:04) come to be missing from the stored list of peers. If they were there then #3394 would stop them being cleaned up, but since they aren't there they will never get cleaned up, and #3394 is not the real problem here.

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 7, 2018

There’s a bug in the way storedPeerList is written back after cleaning up - it can overwrite other peers adding themselves, hence leave them missing from the list.

The removal code should look more like addMyselfTopeerList.

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 7, 2018

Nice, so we have figured it out most of it, right? :-)

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Sep 10, 2018

I wrote a unit test (at #3400) that lets me create and delete hundreds of nodes. This gives me some confidence we understand the main failure mode.

@brb brb added this to the 2.4.1 milestone Sep 12, 2018

@brb brb closed this in fa74e9f Sep 12, 2018

@Raffo

This comment has been minimized.

Copy link
Author

Raffo commented Sep 24, 2018

@bboreham Nice! I've been some time away, but am I right that upgrading to the newly released 2.4.1 should fix the issue? If so I will spin up a cluster and replicate my steps to reproduce the issue.

@bmihaescu

This comment has been minimized.

Copy link

bmihaescu commented Nov 27, 2018

We upgraded to release 2.5.0 and still having this issue.

@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Nov 28, 2018

@bmihaescu please open a new issue with logs so we can diagnose.

@Vlaaaaaaad

This comment has been minimized.

Copy link

Vlaaaaaaad commented Jan 8, 2019

Hi,

We have a very high churn of nodes( a ~75 nodes cluster churning through about 1.000 nodes a day) and after about 2 weeks of running weave 2.5.0 on a kops-deployed 1.10 cluster we got this issue happening again.

We can't share useful logs due to the huge timeframe and the node churn. If you have any idea about how we could share relevant information please let me know.

We basically got @Raffo's commands, put them in a script and have this script run every 3 hours. This solved the issue and we had no more incidents since December.

The relevant part of the script, if anyone needs it:

#!/bin/bash

NODES=$(kubectl get nodes -o template --template='{{range.items}}{{range.status.addresses}}{{if eq .type "InternalIP"}}{{.address}}{{end}}{{end}} {{end}}')

echo Starting NODES cleanup ...
for node in $NODES
do
      #echo $node
      ssh -t -o ConnectTimeout=10 -o StrictHostKeyChecking=no admin@$node "sudo rm /var/lib/weave/weave-netdata.db"
done

echo Starting WEAVE PODS cleanup ...
for weave_pod in $(kubectl get pods -n kube-system | awk '{print $1}' | grep weave)
do
      kubectl delete pod -n kube-system $weave_pod;
done
@bboreham

This comment has been minimized.

Copy link
Member

bboreham commented Jan 8, 2019

Logs from one weave container starting up after you had unreachable nodes and before you ran that script should be useful.

Please open a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.