Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

centos 7.3 with vxlan healthcheck/scheduler/zookeeper-zk always Initializing。 #8496

Closed
ztskycn opened this issue Apr 13, 2017 · 11 comments
Closed

centos 7.3 with vxlan healthcheck/scheduler/zookeeper-zk always Initializing。 #8496

ztskycn opened this issue Apr 13, 2017 · 11 comments
Assignees
@sangeethah
Labels
area/networking kind/bug Issues that are defects reported by users or that we know have reached a real release
Milestone
Release 1.6.0

Comments

@ztskycn
Copy link

ztskycn commented Apr 13, 2017

Describe your issue here

Useful Info
Versions Rancher v1.5.3 Cattle: v0.177.10 UI: v1.5.8
Access localauth admin
Orchestration Cattle
Route stack.index

centos 7.3 with vxlan healthcheck/scheduler/zookeeper-zk always Initializing。
centos 7.3 with ipsec is fine

Uploading image.png…
@niusmallnan
Copy link
Contributor

vxlan: udp 4789
ipsec: udp 500 4500

Pls make sure each agent host can visit these ports.

@ztskycn
Copy link
Author

ztskycn commented Apr 13, 2017

thank you for reply.
all hosts iptables allow input eth0 。

@niusmallnan
Copy link
Contributor

need more information：

  1. healthcheck/metadata/network-manager/scheduler/rancher-server logs
  2. infratructure service package version on your setup.

@ztskycn
Copy link
Author

ztskycn commented Apr 13, 2017

---rancher/healthcheck:v0.2.3
time="2017-04-12T12:00:20Z" level=error msg="Error reading metadata version: Get http://rancher-metadata/2015-12-19/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=5: net/http: request canceled"
time="2017-04-12T21:10:39Z" level=info msg="Scheduling apply config"
time="2017-04-12T21:10:39Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-12T21:10:39Z" level=info msg="Scheduling apply config"
time="2017-04-12T21:10:39Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-12T21:10:39Z" level=info msg="Scheduling apply config"
time="2017-04-12T21:10:39Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-12T21:10:39Z" level=info msg="Scheduling apply config"
time="2017-04-12T21:10:39Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-12T21:10:40Z" level=info msg="Scheduling apply config"
time="2017-04-12T21:10:40Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-12T21:10:40Z" level=info msg="Scheduling apply config"
time="2017-04-12T21:10:40Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-12T21:10:41Z" level=info msg="Scheduling apply config"
time="2017-04-12T21:10:41Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-13T02:48:04Z" level=info msg="Scheduling apply config"
time="2017-04-13T02:48:04Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-13T02:49:09Z" level=info msg="Scheduling apply config"
time="2017-04-13T02:49:09Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-13T02:49:09Z" level=info msg="Scheduling apply config"
time="2017-04-13T02:49:09Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-13T02:49:47Z" level=info msg="Scheduling apply config"
time="2017-04-13T02:49:47Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-13T02:51:00Z" level=info msg="Scheduling apply config"
time="2017-04-13T02:51:01Z" level=info msg="healthCheck -- no changes in haproxy config\n"
time="2017-04-13T02:51:01Z" level=info msg="Scheduling apply config"
time="2017-04-13T02:51:01Z" level=info msg="healthCheck -- no changes in haproxy config\n"

---rancher/metadata:v0.9.1
time="2017-04-13T02:49:09Z" level=info msg="Applied http://192.168.137.164:8080/v1/configcontent/metadata-answers?client=v2&requestedVersion=947?version=947-14a7928f14789d3a00ab33efdfd9c22c"
time="2017-04-13T02:49:09Z" level=info msg="Download and reload in: 40.324947ms"
time="2017-04-13T02:49:09Z" level=info msg="Update requested for version: 948"
time="2017-04-13T02:49:09Z" level=info msg="Downloaded in 20.871933ms"
time="2017-04-13T02:49:09Z" level=info msg="Generating and reloading answers"
time="2017-04-13T02:49:09Z" level=info msg="Generating answers"
time="2017-04-13T02:49:09Z" level=info msg="Generated and reloaded answers"
time="2017-04-13T02:49:09Z" level=info msg="Applied http://192.168.137.164:8080/v1/configcontent/metadata-answers?client=v2&requestedVersion=948?version=948-14a7928f14789d3a00ab33efdfd9c22c"
time="2017-04-13T02:49:09Z" level=info msg="Download and reload in: 34.732185ms"
time="2017-04-13T02:49:47Z" level=info msg="Update requested for version: 949"
time="2017-04-13T02:49:47Z" level=info msg="Downloaded in 24.996582ms"
time="2017-04-13T02:49:47Z" level=info msg="Generating and reloading answers"
time="2017-04-13T02:49:47Z" level=info msg="Generating answers"
time="2017-04-13T02:49:47Z" level=info msg="Generated and reloaded answers"
time="2017-04-13T02:49:47Z" level=info msg="Applied http://192.168.137.164:8080/v1/configcontent/metadata-answers?client=v2&requestedVersion=949?version=949-14a7928f14789d3a00ab33efdfd9c22c"
time="2017-04-13T02:49:47Z" level=info msg="Download and reload in: 54.421392ms"
time="2017-04-13T02:51:00Z" level=info msg="Update requested for version: 950"
time="2017-04-13T02:51:00Z" level=info msg="Downloaded in 21.156098ms"
time="2017-04-13T02:51:00Z" level=info msg="Generating and reloading answers"
time="2017-04-13T02:51:00Z" level=info msg="Generating answers"
time="2017-04-13T02:51:00Z" level=info msg="Generated and reloaded answers"
time="2017-04-13T02:51:00Z" level=info msg="Applied http://192.168.137.164:8080/v1/configcontent/metadata-answers?client=v2&requestedVersion=950?version=950-14a7928f14789d3a00ab33efdfd9c22c"
time="2017-04-13T02:51:00Z" level=info msg="Download and reload in: 39.368367ms"
time="2017-04-13T02:51:00Z" level=info msg="Update requested for version: 951"
time="2017-04-13T02:51:01Z" level=info msg="Downloaded in 18.335544ms"
time="2017-04-13T02:51:01Z" level=info msg="Generating and reloading answers"
time="2017-04-13T02:51:01Z" level=info msg="Generating answers"
time="2017-04-13T02:51:01Z" level=info msg="Generated and reloaded answers"
time="2017-04-13T02:51:01Z" level=info msg="Applied http://192.168.137.164:8080/v1/configcontent/metadata-answers?client=v2&requestedVersion=951?version=951-14a7928f14789d3a00ab33efdfd9c22c"
time="2017-04-13T02:51:01Z" level=info msg="Download and reload in: 29.931669ms"

------rancher/network-manager:v0.6.6
time="2017-04-11T09:09:16Z" level=info msg="Network router changed, syncing ARP tables 2/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:09:29Z" level=info msg="Network router changed, syncing ARP tables 3/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:09:34Z" level=info msg="Network router changed, syncing ARP tables 4/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:09:39Z" level=info msg="Network router changed, syncing ARP tables 5/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:09:44Z" level=info msg="Network router changed, syncing ARP tables 6/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:09:51Z" level=info msg="Network router changed, syncing ARP tables 7/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:09:56Z" level=info msg="Network router changed, syncing ARP tables 8/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:09:56Z" level=info msg="arpsync: (1dad225e55733838b223ebe98c601c4a35db595ebd02f648fcfca72177867323) wrong ARP entry found={LinkIndex:4 Family:2 State:128 Type:1 Flags:0 IP:10.42.185.85 HardwareAddr:0e:00:0a:2a:a8:41}(expected: 02:05:d1:95:4e:a9) for local container, fixing it"
time="2017-04-11T09:10:01Z" level=info msg="Network router changed, syncing ARP tables 9/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:10:06Z" level=info msg="Network router changed, syncing ARP tables 10/10 in containers, new MAC: 02:05:d1:95:4e:a9"
time="2017-04-11T09:10:09Z" level=info msg="CNI down" cid=70c93ddca9fbf77530194800f88d9a3890a86811d494267e986b027b8b5d9c6a networkMode=vxlan
time="2017-04-11T09:10:11Z" level=info msg="Setting up resolv.conf for ContainerId [e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f]"
time="2017-04-11T09:10:11Z" level=info msg="CNI up" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan
time="2017-04-11T09:10:11Z" level=info msg="CNI up done" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan result=IP4:{IP:{IP:10.42.190.71 Mask:ffff0000} Gateway:10.42.0.1 Routes:[{Dst:{IP:169.254.169.250 Mask:ffffffff} GW:} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.42.0.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
net.ipv4.conf.docker0.route_localnet = 1
time="2017-04-11T09:14:30Z" level=info msg="Setting up resolv.conf for ContainerId [e5f33b3f8e0b70126d1860387090f4203dec76ee3572181b20420d4219f35b13]"
time="2017-04-11T09:14:30Z" level=info msg="CNI up" cid=e5f33b3f8e0b70126d1860387090f4203dec76ee3572181b20420d4219f35b13 networkMode=vxlan
time="2017-04-11T09:14:30Z" level=info msg="CNI up done" cid=e5f33b3f8e0b70126d1860387090f4203dec76ee3572181b20420d4219f35b13 networkMode=vxlan result=IP4:{IP:{IP:10.42.79.233 Mask:ffff0000} Gateway:10.42.0.1 Routes:[{Dst:{IP:169.254.169.250 Mask:ffffffff} GW:} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.42.0.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
time="2017-04-11T09:14:30Z" level=info msg="Setting up resolv.conf for ContainerId [34594b5ac4a7de1db03e39251ce049415a5d53d75c6c254fb79334ce1c41766c]"
time="2017-04-11T09:14:30Z" level=info msg="CNI up" cid=34594b5ac4a7de1db03e39251ce049415a5d53d75c6c254fb79334ce1c41766c networkMode=vxlan
time="2017-04-11T09:14:30Z" level=info msg="CNI up done" cid=34594b5ac4a7de1db03e39251ce049415a5d53d75c6c254fb79334ce1c41766c networkMode=vxlan result=IP4:{IP:{IP:10.42.60.195 Mask:ffff0000} Gateway:10.42.0.1 Routes:[{Dst:{IP:169.254.169.250 Mask:ffffffff} GW:} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.42.0.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
time="2017-04-11T21:10:21Z" level=info msg="CNI down" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan
net.ipv4.conf.docker0.route_localnet = 1
time="2017-04-11T21:10:21Z" level=info msg="Setting up resolv.conf for ContainerId [e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f]"
time="2017-04-11T21:10:21Z" level=info msg="CNI up" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan
time="2017-04-11T21:10:21Z" level=info msg="CNI up done" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan result=IP4:{IP:{IP:10.42.190.71 Mask:ffff0000} Gateway:10.42.0.1 Routes:[{Dst:{IP:169.254.169.250 Mask:ffffffff} GW:} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.42.0.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
time="2017-04-12T09:10:30Z" level=info msg="CNI down" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan
net.ipv4.conf.docker0.route_localnet = 1
time="2017-04-12T09:10:30Z" level=info msg="Setting up resolv.conf for ContainerId [e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f]"
time="2017-04-12T09:10:30Z" level=info msg="CNI up" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan
time="2017-04-12T09:10:30Z" level=info msg="CNI up done" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan result=IP4:{IP:{IP:10.42.190.71 Mask:ffff0000} Gateway:10.42.0.1 Routes:[{Dst:{IP:169.254.169.250 Mask:ffffffff} GW:} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.42.0.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
time="2017-04-12T12:00:19Z" level=error msg="Error reading metadata version: Get http://169.254.169.250/2016-07-29/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=120: net/http: request canceled"
time="2017-04-12T12:00:19Z" level=error msg="Error reading metadata version: Get http://169.254.169.250/2016-07-29/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=120: net/http: request canceled"
time="2017-04-12T12:00:20Z" level=error msg="Error reading metadata version: Get http://169.254.169.250/2016-07-29/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=5: net/http: request canceled"
time="2017-04-12T12:00:20Z" level=error msg="Error reading metadata version: Get http://169.254.169.250/2016-07-29/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=5: net/http: request canceled"
time="2017-04-12T12:00:20Z" level=error msg="Error reading metadata version: Get http://169.254.169.250/2016-07-29/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=5: net/http: request canceled"
time="2017-04-12T12:00:20Z" level=error msg="Error reading metadata version: Get http://169.254.169.250/2016-07-29/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=5: net/http: request canceled"
time="2017-04-12T12:00:20Z" level=error msg="Error reading metadata version: Get http://169.254.169.250/2016-07-29/version?wait=true&value=938-14a7928f14789d3a00ab33efdfd9c22c&maxWait=5: net/http: request canceled"
time="2017-04-12T21:10:39Z" level=info msg="CNI down" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan
net.ipv4.conf.docker0.route_localnet = 1
time="2017-04-12T21:10:40Z" level=info msg="Setting up resolv.conf for ContainerId [e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f]"
time="2017-04-12T21:10:40Z" level=info msg="CNI up" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan
time="2017-04-12T21:10:40Z" level=info msg="CNI up done" cid=e2d0a92de2339a26a87e87de3cb2ae5fd584eaa29d04cb6f07648e80ef34ff5f networkMode=vxlan result=IP4:{IP:{IP:10.42.190.71 Mask:ffff0000} Gateway:10.42.0.1 Routes:[{Dst:{IP:169.254.169.250 Mask:ffffffff} GW:} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.42.0.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}
net.ipv4.conf.docker0.route_localnet = 1
time="2017-04-13T02:49:09Z" level=info msg="Setting up resolv.conf for ContainerId [d47d7ebc7679cc2d66a23df53e773d6d4ee76f95ab72393d9a619bf65d757e08]"
time="2017-04-13T02:49:09Z" level=info msg="CNI up" cid=d47d7ebc7679cc2d66a23df53e773d6d4ee76f95ab72393d9a619bf65d757e08 networkMode=vxlan
time="2017-04-13T02:49:09Z" level=info msg="CNI up done" cid=d47d7ebc7679cc2d66a23df53e773d6d4ee76f95ab72393d9a619bf65d757e08 networkMode=vxlan result=IP4:{IP:{IP:10.42.43.188 Mask:ffff0000} Gateway:10.42.0.1 Routes:[{Dst:{IP:169.254.169.250 Mask:ffffffff} GW:} {Dst:{IP:0.0.0.0 Mask:00000000} GW:10.42.0.1}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}

----rancher/scheduler:v0.7.5
time="2017-04-11T09:10:11Z" level=info msg="Connection established"
time="2017-04-11T09:10:11Z" level=info msg="Starting websocket pings"
time="2017-04-11T21:10:21Z" level=error msg="Exiting scheduler with error: "
time="2017-04-11T21:10:22Z" level=info msg="Listening for health checks on 0.0.0.0:80/healthcheck"
time="2017-04-11T21:10:22Z" level=info msg="Connecting to cattle event stream."
time="2017-04-11T21:10:22Z" level=info msg="Subscribing to metadata changes."
time="2017-04-11T21:10:22Z" level=info msg="Initializing event router" workerCount=100
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [instanceReservation] with total 1000000 and used 10 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [cpuReservation] with total 4000 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [memoryReservation] with total 8201961472 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [storageSize] with total 25387704 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [portReservation], ip set [0.0.0.0], ports map tcp map[0.0.0.0:map[]], ports map udp map[0.0.0.0:map[4789:251c5c52-8205-4e75-8d39-79699c0bf885]] for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [hostLabels] with label map [map[io.rancher.host.agent_image:rancher/agent:v1.2.1 io.rancher.host.docker_version:1.10 io.rancher.host.linux_kernel_version:3.10]]"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [instanceReservation] with total 1000000 and used 8 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [cpuReservation] with total 4000 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [memoryReservation] with total 8201961472 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [storageSize] with total 25387290 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-11T21:10:22Z" level=info msg="Connection established"
time="2017-04-11T21:10:22Z" level=info msg="Starting websocket pings"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [portReservation], ip set [0.0.0.0], ports map tcp map[0.0.0.0:map[]], ports map udp map[0.0.0.0:map[4789:7b5a75ee-5c62-4c33-a40a-5e2feecfcaec]] for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-11T21:10:22Z" level=info msg="Adding resource pool [hostLabels] with label map [map[io.rancher.host.agent_image:rancher/agent:v1.2.1 io.rancher.host.docker_version:1.10 io.rancher.host.linux_kernel_version:3.10]]"
time="2017-04-12T09:10:30Z" level=error msg="Exiting scheduler with error: "
time="2017-04-12T09:10:30Z" level=info msg="Listening for health checks on 0.0.0.0:80/healthcheck"
time="2017-04-12T09:10:30Z" level=info msg="Subscribing to metadata changes."
time="2017-04-12T09:10:30Z" level=info msg="Connecting to cattle event stream."
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [instanceReservation] with total 1000000 and used 10 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [cpuReservation] with total 4000 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [memoryReservation] with total 8201961472 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [storageSize] with total 25387704 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T09:10:31Z" level=info msg="Initializing event router" workerCount=100
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [portReservation], ip set [0.0.0.0], ports map tcp map[0.0.0.0:map[]], ports map udp map[0.0.0.0:map[4789:251c5c52-8205-4e75-8d39-79699c0bf885]] for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [hostLabels] with label map [map[io.rancher.host.agent_image:rancher/agent:v1.2.1 io.rancher.host.docker_version:1.10 io.rancher.host.linux_kernel_version:3.10]]"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [cpuReservation] with total 4000 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [memoryReservation] with total 8201961472 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [storageSize] with total 25387290 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [instanceReservation] with total 1000000 and used 8 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [portReservation], ip set [0.0.0.0], ports map tcp map[0.0.0.0:map[]], ports map udp map[0.0.0.0:map[4789:7b5a75ee-5c62-4c33-a40a-5e2feecfcaec]] for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T09:10:31Z" level=info msg="Adding resource pool [hostLabels] with label map [map[io.rancher.host.agent_image:rancher/agent:v1.2.1 io.rancher.host.docker_version:1.10 io.rancher.host.linux_kernel_version:3.10]]"
time="2017-04-12T09:10:31Z" level=info msg="Connection established"
time="2017-04-12T09:10:31Z" level=info msg="Starting websocket pings"
time="2017-04-12T21:10:39Z" level=error msg="Exiting scheduler with error: "
time="2017-04-12T21:10:40Z" level=info msg="Listening for health checks on 0.0.0.0:80/healthcheck"
time="2017-04-12T21:10:40Z" level=info msg="Connecting to cattle event stream."
time="2017-04-12T21:10:40Z" level=info msg="Subscribing to metadata changes."
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [memoryReservation] with total 8201961472 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [storageSize] with total 25387704 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [instanceReservation] with total 1000000 and used 10 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [cpuReservation] with total 4000 and used 0 for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T21:10:40Z" level=info msg="Initializing event router" workerCount=100
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [portReservation], ip set [0.0.0.0], ports map tcp map[0.0.0.0:map[]], ports map udp map[0.0.0.0:map[4789:251c5c52-8205-4e75-8d39-79699c0bf885]] for host 64e55d49-d006-48ef-9ffd-cf5def74eb09"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [hostLabels] with label map [map[io.rancher.host.agent_image:rancher/agent:v1.2.1 io.rancher.host.docker_version:1.10 io.rancher.host.linux_kernel_version:3.10]]"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [memoryReservation] with total 8201961472 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [storageSize] with total 25387290 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [instanceReservation] with total 1000000 and used 8 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [cpuReservation] with total 4000 and used 0 for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T21:10:40Z" level=info msg="Connection established"
time="2017-04-12T21:10:40Z" level=info msg="Starting websocket pings"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [portReservation], ip set [0.0.0.0], ports map tcp map[0.0.0.0:map[]], ports map udp map[0.0.0.0:map[4789:7b5a75ee-5c62-4c33-a40a-5e2feecfcaec]] for host 6c997f27-c535-40eb-bb19-2098fccd1d60"
time="2017-04-12T21:10:40Z" level=info msg="Adding resource pool [hostLabels] with label map [map[io.rancher.host.agent_image:rancher/agent:v1.2.1 io.rancher.host.docker_version:1.10 io.rancher.host.linux_kernel_version:3.10]]"

----rancher/server:1.5.3
time="2017-04-11T09:13:55Z" level=info msg="[zookeeper:zk-conf]: Created " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:13:55Z" level=info msg="[zookeeper:]: Project created " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:13:55Z" level=info msg="[zookeeper:]: Starting project " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:13:55Z" level=info msg="[zookeeper:zk]: Starting " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:14:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T09:14:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T09:14:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T09:14:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T09:14:43Z" level=info msg="[zookeeper:zk]: Started " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:14:43Z" level=info msg="[zookeeper:zk-conf]: Starting " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:14:43Z" level=info msg="[zookeeper:zk-conf]: Started " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:14:43Z" level=info msg="[zookeeper:zk-volume]: Starting " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:14:43Z" level=info msg="[zookeeper:zk-volume]: Started " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:14:43Z" level=info msg="[zookeeper:]: Project started " eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T09:14:43Z" level=info msg="Stack Create Event Done" eventId=5ead64c4-0980-4e87-a1d8-5c09f05a939d resourceId=1st17
time="2017-04-11T11:38:49Z" level=info msg="Exiting rancher-compose-executor" version=v0.13.1
time="2017-04-11T11:38:50Z" level=info msg="Starting rancher-compose-executor" version=v0.13.1
time="2017-04-11T11:38:51Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-11T11:38:51Z" level=info msg="Connection established"
time="2017-04-11T11:38:51Z" level=info msg="Starting websocket pings"
time="2017-04-11T11:38:53Z" level=info msg="Exiting go-machine-service"
time="2017-04-11T11:38:54Z" level=info msg="Setting log level" logLevel=info
time="2017-04-11T11:38:54Z" level=info msg="Starting go-machine-service..." gitcommit=v0.36.1
time="2017-04-11T11:38:54Z" level=info msg="Waiting for handler registration (1/2)"
time="2017-04-11T11:38:55Z" level=info msg="Initializing event router" workerCount=250
time="2017-04-11T11:38:55Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-11T11:38:55Z" level=info msg="Connection established"
time="2017-04-11T11:38:55Z" level=info msg="Starting websocket pings"
time="2017-04-11T11:38:55Z" level=info msg="Waiting for handler registration (2/2)"
time="2017-04-11T11:38:55Z" level=info msg="Connection established"
time="2017-04-11T11:38:55Z" level=info msg="Starting websocket pings"
time="2017-04-11T11:38:55Z" level=info msg="Installing builtin drivers"
time="2017-04-11T11:38:55Z" level=info msg="Downloading all drivers"
time="2017-04-11T11:38:55Z" level=info msg="Done downloading all drivers"
time="2017-04-11T11:40:14Z" level=info msg="Shutting down backend ffde4a6f-9fed-4cc5-483f-419ca2124384. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-11T11:40:14Z" level=info msg="Removed backend. Key: ffde4a6f-9fed-4cc5-483f-419ca2124384. Session ID 21f87e0b-b84d-4b79-a7c8-65d566a90486 ."
time="2017-04-11T11:40:20Z" level=info msg="Handling backend connection request."
time="2017-04-11T11:40:20Z" level=info msg="Registering backend for host ffde4a6f-9fed-4cc5-483f-419ca2124384 with session ID b0c0521b-f71d-4fb8-8a1f-e115bd613b71."
time="2017-04-11T12:39:56Z" level=info msg="Couldn't find frontend channel for key ffde4a6f-9fed-4cc5-483f-419ca2124384. Closing frontend connection."
time="2017-04-11T16:39:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T16:39:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T16:39:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T16:39:27Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T18:54:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T18:54:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T18:54:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T18:54:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T19:25:42Z" level=info msg="Shutting down backend 48a7011f-adc5-4826-52df-2e16d3af28a6. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-11T19:25:42Z" level=info msg="Removed backend. Key: 48a7011f-adc5-4826-52df-2e16d3af28a6. Session ID a6f49a6b-0fa1-45eb-9dad-eafda1374fdd ."
time="2017-04-11T19:25:49Z" level=info msg="Handling backend connection request."
time="2017-04-11T19:25:49Z" level=info msg="Registering backend for host 48a7011f-adc5-4826-52df-2e16d3af28a6 with session ID 96c61b3a-d367-40c7-8ac5-43d28e9b1a68."
time="2017-04-11T19:59:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T19:59:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T19:59:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T19:59:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-11T21:06:52Z" level=info msg="Shutting down backend 40173342-0296-4234-7467-e4f1de3fd46f. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-11T21:06:52Z" level=info msg="Removed backend. Key: 40173342-0296-4234-7467-e4f1de3fd46f. Session ID 3ec32980-5721-43ba-ac54-e35dc8968afe ."
time="2017-04-11T21:06:57Z" level=info msg="Handling backend connection request."
time="2017-04-11T21:06:57Z" level=info msg="Registering backend for host 40173342-0296-4234-7467-e4f1de3fd46f with session ID 40a9ccef-d0a8-4698-ae66-22836e66d6e2."
time="2017-04-11T21:07:10Z" level=info msg="Shutting down backend 65e0589b-d00c-4667-6b03-923fcd26e5cb. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-11T21:07:10Z" level=info msg="Removed backend. Key: 65e0589b-d00c-4667-6b03-923fcd26e5cb. Session ID 089f4028-a916-4ced-99a5-b9a9a76f6854 ."
time="2017-04-11T21:07:15Z" level=info msg="Handling backend connection request."
time="2017-04-11T21:07:15Z" level=info msg="Registering backend for host 65e0589b-d00c-4667-6b03-923fcd26e5cb with session ID 5c9e019c-be97-4dcd-ba02-af255cc76a4f."
time="2017-04-11T23:38:59Z" level=info msg="Exiting rancher-compose-executor" version=v0.13.1
time="2017-04-11T23:39:01Z" level=info msg="Starting rancher-compose-executor" version=v0.13.1
time="2017-04-11T23:39:02Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-11T23:39:02Z" level=info msg="Connection established"
time="2017-04-11T23:39:02Z" level=info msg="Starting websocket pings"
time="2017-04-11T23:39:02Z" level=info msg="Exiting go-machine-service"
time="2017-04-11T23:39:03Z" level=info msg="Setting log level" logLevel=info
time="2017-04-11T23:39:03Z" level=info msg="Starting go-machine-service..." gitcommit=v0.36.1
time="2017-04-11T23:39:03Z" level=info msg="Waiting for handler registration (1/2)"
time="2017-04-11T23:39:04Z" level=info msg="Initializing event router" workerCount=250
time="2017-04-11T23:39:04Z" level=info msg="Connection established"
time="2017-04-11T23:39:04Z" level=info msg="Starting websocket pings"
time="2017-04-11T23:39:04Z" level=info msg="Waiting for handler registration (2/2)"
time="2017-04-11T23:39:08Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-11T23:39:08Z" level=info msg="Connection established"
time="2017-04-11T23:39:08Z" level=info msg="Starting websocket pings"
time="2017-04-11T23:39:08Z" level=info msg="Installing builtin drivers"
time="2017-04-11T23:39:09Z" level=info msg="Downloading all drivers"
time="2017-04-11T23:39:09Z" level=info msg="Done downloading all drivers"
time="2017-04-11T23:40:27Z" level=info msg="Shutting down backend ffde4a6f-9fed-4cc5-483f-419ca2124384. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-11T23:40:27Z" level=info msg="Removed backend. Key: ffde4a6f-9fed-4cc5-483f-419ca2124384. Session ID b0c0521b-f71d-4fb8-8a1f-e115bd613b71 ."
time="2017-04-11T23:40:33Z" level=info msg="Handling backend connection request."
time="2017-04-11T23:40:33Z" level=info msg="Registering backend for host ffde4a6f-9fed-4cc5-483f-419ca2124384 with session ID 4ba33d4d-103a-4fdb-a6da-ba4c2afc73bc."
time="2017-04-12T05:09:19Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-12T05:33:30Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-12T07:25:59Z" level=info msg="Shutting down backend 48a7011f-adc5-4826-52df-2e16d3af28a6. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T07:25:59Z" level=info msg="Removed backend. Key: 48a7011f-adc5-4826-52df-2e16d3af28a6. Session ID 96c61b3a-d367-40c7-8ac5-43d28e9b1a68 ."
time="2017-04-12T07:26:04Z" level=info msg="Handling backend connection request."
time="2017-04-12T07:26:04Z" level=info msg="Registering backend for host 48a7011f-adc5-4826-52df-2e16d3af28a6 with session ID 57aa2b22-5a3f-49c6-930f-06967c1fdc49."
time="2017-04-12T09:07:06Z" level=info msg="Shutting down backend 40173342-0296-4234-7467-e4f1de3fd46f. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T09:07:06Z" level=info msg="Removed backend. Key: 40173342-0296-4234-7467-e4f1de3fd46f. Session ID 40a9ccef-d0a8-4698-ae66-22836e66d6e2 ."
time="2017-04-12T09:07:11Z" level=info msg="Handling backend connection request."
time="2017-04-12T09:07:11Z" level=info msg="Registering backend for host 40173342-0296-4234-7467-e4f1de3fd46f with session ID e94b3998-9e57-44ce-8220-50982463784c."
time="2017-04-12T09:07:24Z" level=info msg="Shutting down backend 65e0589b-d00c-4667-6b03-923fcd26e5cb. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T09:07:24Z" level=info msg="Removed backend. Key: 65e0589b-d00c-4667-6b03-923fcd26e5cb. Session ID 5c9e019c-be97-4dcd-ba02-af255cc76a4f ."
time="2017-04-12T09:07:29Z" level=info msg="Handling backend connection request."
time="2017-04-12T09:07:29Z" level=info msg="Registering backend for host 65e0589b-d00c-4667-6b03-923fcd26e5cb with session ID 278e9062-e7fd-4e1c-b357-a149735c63ed."
time="2017-04-12T11:39:11Z" level=info msg="Exiting rancher-compose-executor" version=v0.13.1
time="2017-04-12T11:39:12Z" level=info msg="Starting rancher-compose-executor" version=v0.13.1
time="2017-04-12T11:39:13Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-12T11:39:13Z" level=info msg="Connection established"
time="2017-04-12T11:39:13Z" level=info msg="Starting websocket pings"
time="2017-04-12T11:39:14Z" level=info msg="Exiting go-machine-service"
time="2017-04-12T11:39:14Z" level=info msg="Setting log level" logLevel=info
time="2017-04-12T11:39:14Z" level=info msg="Starting go-machine-service..." gitcommit=v0.36.1
time="2017-04-12T11:39:14Z" level=info msg="Waiting for handler registration (1/2)"
time="2017-04-12T11:39:15Z" level=info msg="Initializing event router" workerCount=250
time="2017-04-12T11:39:15Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-12T11:39:15Z" level=info msg="Connection established"
time="2017-04-12T11:39:15Z" level=info msg="Starting websocket pings"
time="2017-04-12T11:39:15Z" level=info msg="Waiting for handler registration (2/2)"
time="2017-04-12T11:39:15Z" level=info msg="Connection established"
time="2017-04-12T11:39:15Z" level=info msg="Starting websocket pings"
time="2017-04-12T11:39:15Z" level=info msg="Installing builtin drivers"
time="2017-04-12T11:39:15Z" level=info msg="Downloading all drivers"
time="2017-04-12T11:39:15Z" level=info msg="Done downloading all drivers"
time="2017-04-12T11:40:42Z" level=info msg="Shutting down backend ffde4a6f-9fed-4cc5-483f-419ca2124384. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T11:40:42Z" level=info msg="Removed backend. Key: ffde4a6f-9fed-4cc5-483f-419ca2124384. Session ID 4ba33d4d-103a-4fdb-a6da-ba4c2afc73bc ."
time="2017-04-12T11:40:47Z" level=info msg="Handling backend connection request."
time="2017-04-12T11:40:47Z" level=info msg="Registering backend for host ffde4a6f-9fed-4cc5-483f-419ca2124384 with session ID bc97179b-6db0-4c3b-8290-e30b3fe8bd7e."
time="2017-04-12T18:24:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-12T18:24:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-12T18:24:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-12T18:24:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-12T19:26:12Z" level=info msg="Shutting down backend 48a7011f-adc5-4826-52df-2e16d3af28a6. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T19:26:12Z" level=info msg="Removed backend. Key: 48a7011f-adc5-4826-52df-2e16d3af28a6. Session ID 57aa2b22-5a3f-49c6-930f-06967c1fdc49 ."
time="2017-04-12T19:26:18Z" level=info msg="Handling backend connection request."
time="2017-04-12T19:26:18Z" level=info msg="Registering backend for host 48a7011f-adc5-4826-52df-2e16d3af28a6 with session ID 8c9c7d91-1b6e-499a-b560-85f2b929b10a."
time="2017-04-12T21:07:18Z" level=info msg="Shutting down backend 40173342-0296-4234-7467-e4f1de3fd46f. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T21:07:18Z" level=info msg="Removed backend. Key: 40173342-0296-4234-7467-e4f1de3fd46f. Session ID e94b3998-9e57-44ce-8220-50982463784c ."
time="2017-04-12T21:07:24Z" level=info msg="Handling backend connection request."
time="2017-04-12T21:07:24Z" level=info msg="Registering backend for host 40173342-0296-4234-7467-e4f1de3fd46f with session ID 963ae1ba-2a62-49df-8150-3f778fe72319."
time="2017-04-12T21:07:37Z" level=info msg="Shutting down backend 65e0589b-d00c-4667-6b03-923fcd26e5cb. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T21:07:37Z" level=info msg="Removed backend. Key: 65e0589b-d00c-4667-6b03-923fcd26e5cb. Session ID 278e9062-e7fd-4e1c-b357-a149735c63ed ."
time="2017-04-12T21:07:43Z" level=info msg="Handling backend connection request."
time="2017-04-12T21:07:43Z" level=info msg="Registering backend for host 65e0589b-d00c-4667-6b03-923fcd26e5cb with session ID f12933e9-f3fd-4bc9-81be-40651bcb0f46."
time="2017-04-12T23:39:19Z" level=info msg="Exiting rancher-compose-executor" version=v0.13.1
time="2017-04-12T23:39:21Z" level=info msg="Starting rancher-compose-executor" version=v0.13.1
time="2017-04-12T23:39:22Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-12T23:39:22Z" level=info msg="Connection established"
time="2017-04-12T23:39:22Z" level=info msg="Starting websocket pings"
time="2017-04-12T23:39:22Z" level=info msg="Exiting go-machine-service"
time="2017-04-12T23:39:23Z" level=info msg="Setting log level" logLevel=info
time="2017-04-12T23:39:23Z" level=info msg="Starting go-machine-service..." gitcommit=v0.36.1
time="2017-04-12T23:39:23Z" level=info msg="Waiting for handler registration (1/2)"
time="2017-04-12T23:39:24Z" level=info msg="Initializing event router" workerCount=10
time="2017-04-12T23:39:24Z" level=info msg="Initializing event router" workerCount=250
time="2017-04-12T23:39:24Z" level=info msg="Connection established"
time="2017-04-12T23:39:24Z" level=info msg="Connection established"
time="2017-04-12T23:39:24Z" level=info msg="Starting websocket pings"
time="2017-04-12T23:39:24Z" level=info msg="Starting websocket pings"
time="2017-04-12T23:39:24Z" level=info msg="Waiting for handler registration (2/2)"
time="2017-04-12T23:39:24Z" level=info msg="Installing builtin drivers"
time="2017-04-12T23:39:24Z" level=info msg="Downloading all drivers"
time="2017-04-12T23:39:24Z" level=info msg="Done downloading all drivers"
time="2017-04-12T23:40:54Z" level=info msg="Shutting down backend ffde4a6f-9fed-4cc5-483f-419ca2124384. Connection closed because: websocket: close 1006 unexpected EOF."
time="2017-04-12T23:40:54Z" level=info msg="Removed backend. Key: ffde4a6f-9fed-4cc5-483f-419ca2124384. Session ID bc97179b-6db0-4c3b-8290-e30b3fe8bd7e ."
time="2017-04-12T23:41:00Z" level=info msg="Handling backend connection request."
time="2017-04-12T23:41:00Z" level=info msg="Registering backend for host ffde4a6f-9fed-4cc5-483f-419ca2124384 with session ID c2c57ede-a792-479a-be1f-ba662b19e6da."
time="2017-04-13T00:04:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-13T00:04:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-13T00:04:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-13T00:04:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-13T00:09:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-13T00:09:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-13T00:09:26Z" level=error msg="Failed to update existing repo cache: exit status 128"
time="2017-04-13T00:09:26Z" level=error msg="Failed to update existing repo cache: exit status 128"

@niusmallnan
Copy link
Contributor

After remove labels on vxlan service:
io.rancher.network.macsync: 'true'
io.rancher.network.arpsync: 'true'

Healthcheck can work fine, but I think there may be some bugs on network-manager.
Macsync and arpsync are new features for network-manager.

To Be Continued...

@niusmallnan
Copy link
Contributor

Just do some diagnosis, I think I have found the root cause.

If we add arpsync label for vxlan, network-manager will update arp table in containers.
For vxlan, we know that it use vtep1042 for cross-host communication.

In normally, arp table shoud use vtep1042 mac address.
image

If we add arpsync, arp table will update as below.
image

It should use vtep1042 mac address, not vxlan-eth0 mac address.

I can reproduce this issue very easy everywhere！

@leodotcloud
Copy link
Collaborator

@niusmallnan Yes it's a bug, need to skip vtep interface for ARP sync.

@ztskycn ztskycn closed this as completed Apr 14, 2017
@ztskycn ztskycn reopened this Apr 15, 2017
@leodotcloud leodotcloud mentioned this issue Apr 17, 2017
Merged
@leodotcloud leodotcloud self-assigned this Apr 17, 2017
@leodotcloud leodotcloud added area/networking kind/bug Issues that are defects reported by users or that we know have reached a real release labels Apr 17, 2017
@niusmallnan niusmallnan mentioned this issue Apr 21, 2017
Closed
@deniseschannon
Copy link

Fixed in network-manager:v0.7.0 in network-services:v0.2.0 in the v1.6.0-dev branch

@deniseschannon deniseschannon added this to the Release 1.6.0 milestone Apr 26, 2017
@thaoula
Copy link

thaoula commented Apr 26, 2017

Hi Guys,

Any chance this can be released as a patch for 1.5.6 or as 1.5.7. Currently, it seems we can only run a single node.

I have tried setting up multiple nodes on Azure, Vsphere, Parallels all get same issue.. Health Check and Scheduler all stuck initializing.

This issue existed in 1.5.5 also.

When is 1.6.0 due?

Regards,
Tarek

@deniseschannon
Copy link

@thaoula We are running some tests to see if users can v1.5.x can use the updated network-manager image without pushing a new template. We can't push out a new template to v1.5.x due to an automatic upgrade of network-services to all users (not just users using vxlan). If so, we will recommend that path for v1.5.x. I'll keep you posted after we do some internal testing

Otherwise, v1.6.0 is targeted for end of month.

@sangeethah sangeethah assigned sangeethah and unassigned cjellick Apr 27, 2017
@sangeethah
Copy link
Contributor

sangeethah commented Apr 27, 2017 •
edited

Tested with network-manager:v0.7.0 in network-services:v0.2.0 in the v1.6.0-dev branch using master builds.

This issue is not seen anymore. healthcheck/scheduler and other health check enabled instances get to "healthy" state as expected.

@leodotcloud leodotcloud mentioned this issue May 2, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sangeethah sangeethah

Labels
area/networking kind/bug Issues that are defects reported by users or that we know have reached a real release
Projects
None yet
Milestone
Release 1.6.0
Development

No branches or pull requests
7 participants
@thaoula @niusmallnan @cjellick @sangeethah @deniseschannon @ztskycn @leodotcloud