Possible bug on v0.1.0rc1 - Can't find configuration for a new passive connection from..

[tonidas]

Hey guys, here at Inloco we faced an issue with the brand new v0.1.0rc1, using KOPS and CentOS 7.

As soon as we updated, we started to see pods that were unable to connect with other services around the kubernetes cluster. In this issue I will call the kube-route pods of the nodes where the pods were facing network problems as "weird kube-route pods".

Also, some nodes started to became unreachable by SSH and other services, therefore with the "NotReady" state. In this issue I will call the NotReady nodes as "unreachable nodes".

On the weird kube-route pods, we saw some new lines appearing periodically on the logs, like this one:
time="2018-01-05T00:22:42Z" level=info msg="Can't find configuration for a new passive connection from:10.0.62.85" Topic=Peer

10.0.62.X is the pattern of the internal ips of this kubernetes cluster, therefore it seems to be a problem on the connection with neighbors on the BGP.

Running gobgp neighbor on the weird kube-routes, we saw an empty list of neighbors:

Peer AS Up/Down State |#Received Accepted

Debugging with TCPdump, we saw that every "BGP OPEN" started by other healthy kube-route to a weird one was responded with an TCP ACK, followed by a TCP [RST, ACK]. A behaviour that simply doesn't happen on healthy kube-route pods receiving "BGP OPEN".
No. Time Src Dest Protocol Info
82 0.552669 10.0.62.115 10.0.62.155 BGP OPEN Message
83 0.552683 10.0.62.155 10.0.62.115 TCP 179 → 35216 [ACK] Seq=1 Ack=46 Win=27136 Len=0 TSval=5750002 TSecr=947513
84 0.552730 10.0.62.155 10.0.62.115 TCP 179 → 35216 [RST, ACK] Seq=1 Ack=46 Win=27136 Len=0 TSval=0 TSecr=947513

Running gobgp neighbor on the healthier kube-router pods, it seems that the weird kube-routers keep changing the state from Active to Idle, and never reaching the Established state.

Peer AS Up/Down State |#Received Accepted
10.0.62.12 64512 00:24:08 Establ | 1 1
10.0.62.14 64512 00:24:13 Establ | 1 1
10.0.62.21 64512 never Active | 0 0
10.0.62.28 64512 never Active | 0 0
10.0.62.31 64512 never Active | 0 0
10.0.62.40 64512 never Active | 0 0
10.0.62.46 64512 00:24:08 Establ | 1 1
10.0.62.47 64512 00:24:12 Establ | 1 1
10.0.62.58 64512 00:24:15 Establ | 1 1
10.0.62.72 64512 00:03:27 Establ | 1 1
10.0.62.76 64512 00:13:41 Establ | 1 1
10.0.62.88 64512 00:24:07 Establ | 1 1
10.0.62.97 64512 00:05:41 Active | 0 0
10.0.62.100 64512 00:03:40 Establ | 1 1
10.0.62.104 64512 never Active | 0 0
10.0.62.106 64512 never Active | 0 0
10.0.62.121 64512 00:24:07 Establ | 1 1
10.0.62.133 64512 00:24:10 Establ | 1 1
10.0.62.142 64512 00:24:11 Establ | 1 1

Regarding the unreachable nodes, here what the system logs show to us:

[ 54.462943] cloud-init[2091]: Cloud-init v. 0.7.5 finished at Thu, 04 Jan 2018 21:24:17 +0000. Datasource DataSourceEc2. Up 54.45 seconds
[�[32m OK �[0m] Started Execute cloud user/final scripts.
[�[32m OK �[0m] Reached target Multi-User System.
Starting Update UTMP about System Runlevel Changes...
[�[32m OK �[0m] Started Update UTMP about System Runlevel Changes.
[ 56.530099] TECH PREVIEW: Overlay filesystem may not be fully supported.
[ 56.530099] Please review provided documentation for limitations.
[ 88.297253] Netfilter messages via NETLINK v0.30.
[ 88.310799] ip_set: protocol 6
[ 88.410042] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[ 88.419135] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[ 88.430460] IPVS: Creating netns size=2040 id=0
[ 88.438116] IPVS: ipvs loaded.
[ 88.516456] IPVS: [rr] scheduler registered.
[ 99.749934] IPVS: Creating netns size=2040 id=1
[ 99.930871] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 99.962226] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 99.997493] device veth28020ad5 entered promiscuous mode
[ 100.012649] kube-bridge: port 1(veth28020ad5) entered forwarding state
[ 100.043608] kube-bridge: port 1(veth28020ad5) entered forwarding state
[ 108.054266] kube-bridge: port 1(veth28020ad5) entered disabled state
[ 108.074183] device veth28020ad5 left promiscuous mode
[ 108.083412] kube-bridge: port 1(veth28020ad5) entered disabled state
[ 160.471316] IPVS: Creating netns size=2040 id=2
[ 160.608197] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 160.635159] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 160.682412] device veth86083d0e entered promiscuous mode
[ 160.693758] kube-bridge: port 1(veth86083d0e) entered forwarding state
[ 160.701187] IPVS: Creating netns size=2040 id=3
[ 160.713732] kube-bridge: port 1(veth86083d0e) entered forwarding state
[ 160.845818] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 160.871921] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 160.909392] device veth05103c95 entered promiscuous mode
[ 160.916303] kube-bridge: port 2(veth05103c95) entered forwarding state
[ 160.924690] kube-bridge: port 2(veth05103c95) entered forwarding state
[ 2796.646752] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2796.935848] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2797.221681] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2797.235676] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2797.303722] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2797.389171] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2797.405837] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2797.681278] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2797.905382] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2806.285303] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2806.647101] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2806.935176] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2807.221333] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2807.236081] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2807.303322] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2807.389733] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2807.405855] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2807.681926] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2807.905011] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2816.285718] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2816.646471] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2816.935530] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2817.221725] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2817.236536] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2817.303896] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2817.390027] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2817.405808] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2817.681284] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2817.905444] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2826.286211] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2826.647024] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2826.935147] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2827.221185] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2827.236340] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2827.303373] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2827.389672] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2827.405744] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2827.682117] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2827.905094] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2836.285862] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2836.646669] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2836.935815] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2837.221251] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2837.235725] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2837.303991] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2837.389298] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2837.390272] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2837.681804] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2837.905694] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2846.286267] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2846.647051] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2846.935475] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2847.221772] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2847.236412] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2847.303661] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2847.389871] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2847.406314] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2847.681352] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2847.905958] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2856.285575] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2856.646678] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2856.935894] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2857.221204] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2857.236080] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2857.303610] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2857.389290] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2857.420064] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2857.681897] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2857.905538] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2866.286051] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2866.646775] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2866.935341] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2867.221290] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2867.236565] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2867.304054] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2867.389841] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2867.407849] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2867.681264] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2867.905059] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2876.285444] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2876.647149] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2876.935799] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2877.221792] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2877.235933] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2877.303604] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2877.389433] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2877.406222] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2877.681899] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2877.905320] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2886.285756] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2886.647123] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2886.935288] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2887.221448] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2887.236316] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2887.303399] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2887.389998] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2887.406332] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2887.681539] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2887.905741] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2896.286292] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2896.647102] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2896.936027] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2897.221786] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2897.235854] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2897.303965] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2897.389747] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2897.405592] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2897.682021] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2897.905350] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2906.285528] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2906.646452] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2906.935129] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2907.221430] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2907.236310] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2907.303697] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2907.389323] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2907.406288] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2907.681455] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2907.905748] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2916.285913] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2916.646931] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2916.935511] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2917.303407] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2917.389847] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2917.405613] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2917.682133] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2917.905416] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2926.286380] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2926.647199] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2926.935152] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2927.304160] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2927.389237] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2927.406219] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2927.681301] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2927.905725] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2936.285672] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2936.646321] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2936.936159] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2937.303603] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2937.389636] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2937.406304] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2937.681754] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2937.906020] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2946.285904] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2946.646816] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2946.935729] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2947.303550] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2947.389309] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2947.405438] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2947.681216] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2947.905628] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2956.286247] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2956.646449] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2956.935467] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2957.303982] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2957.389699] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2957.405680] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2957.681956] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2957.905099] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2966.286045] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2966.647014] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2966.935642] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2967.303824] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2967.389411] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2967.405671] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2967.682170] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2967.905547] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2976.286351] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2976.646798] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2976.935104] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2977.303546] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2977.389899] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2977.411811] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2977.681526] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2977.905238] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2986.285826] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2986.646396] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2986.935454] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2987.303963] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2987.389407] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2987.405549] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2987.681342] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2987.905849] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2996.286236] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2996.647130] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 2996.935867] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2997.221918] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2997.236606] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 2997.303749] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2997.389900] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 2997.405418] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 2997.681943] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 2997.905139] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 3006.285718] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 3006.646322] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 3006.935475] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 3007.221611] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 3007.236206] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 3007.303498] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 3007.389564] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 3007.405490] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 3007.681318] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 3007.905602] IPVS: rr: TCP 10.0.62.39:30184 - no destination available
[ 3016.286161] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 3016.646769] IPVS: rr: TCP 10.0.62.39:32739 - no destination available
[ 3016.935244] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 3017.221249] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 3017.235939] IPVS: rr: TCP 10.0.62.39:30109 - no destination available
[ 3017.303929] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 3017.390118] IPVS: rr: TCP 10.0.62.39:30023 - no destination available
[ 3017.406323] IPVS: rr: TCP 10.0.62.39:31422 - no destination available
[ 3017.681369] IPVS: rr: TCP 10.0.62.39:30184 - no destination available

Because of those problems, we downgraded the version to v0.0.20.
We upgraded "automatically" because KOPS seems to use the latest kube-route version on the DaemonSet, triggering a magic update for new nodes after the docker hub tag "latest" update (kubernetes/kops#4196).

[murali-reddy]

@tonidas thans for reporting .

I figured the root cause. Perhaps this may be specific to KOPS. DS spec in kops for the kube-router is updated to use InitContainers field of pod spec instead of annotation. There were some issue with upgrades like for e.g kubernetes/kops#3986

But in general something weird is happening with initContainers field specified as part of pod spec. In kube-router initcontainer populates the CIDR for the node in CNI conf. It seems the kube-router container might have started before init-containet set the cni conf.

failing with error “E0105 06:33:39.335630       1 network_routes_controller.go:85] Failed to get pod CIDR from CNI conf file: subnet missing from CNI IPAM”

there is timing mismatch between init container and kube-router container. For now i will add a work-around.

But i dont think its particular to new release v0.1.0rc1, it just manifested during the upgrade.

[Cisneiros]

Hi @murali-reddy! Thanks for your reply and for your work on kube-router. I work with @tonidas, and we were debugging this last night. What I can add to the discussion is:

• We haven't had this problem before yesterday (when the new version was released).
• We recreated all kube-router pods multiple times during the day, trying to solve this, and the issue always came back after a few minutes (or half hour, once or twice)
• We created a brand new cluster (using kops), using the new kube-router version. This cluster presented the same issue, and we deleted the pods multiple times on that new cluster as well. same results.
• We reverted the version on both the new (clean) cluster, and on the previous (heavy load) cluster. The symptoms disappeared on the first try and didn't come back.

From our observations, it does seem likely the new version contributes for the issue. Of course, it could be a coincidence. I hope this helps investigating!

[andrewsykim]

fwiw, using latest tag in kops should probably be addressed (https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.kuberouter/k8s-1.6.yaml.template#L40)

[Cisneiros]

For sure! We opened an issue there as well: kubernetes/kops#4196

[andrewsykim]

Awesome, thanks!!

[murali-reddy]

@Cisneiros i was able to reproduce (2 out of 5 fresh clusters) this issue. Strange, but now I am not able to reproduce at all. If you have a cluster with the issue or if you can reproduce this issue, can you confirm the symptoms i observed above. i.e) On a node which is not BGP peered with other nodes, if you see kube-router logs do you see error like below?

Failed to get pod CIDR from CNI conf file: subnet missing from CNI IPAM”

[murali-reddy]

But in general something weird is happening with initContainers field specified as part of pod spec. In kube-router initcontainer populates the CIDR for the node in CNI conf. It seems the kube-router container might have started before init-containet set the cni conf.

failing with error “E0105 06:33:39.335630       1 network_routes_controller.go:85] Failed to get pod CIDR from CNI conf file: subnet missing from CNI IPAM”
there is timing mismatch between init container and kube-router container. For now i will add a work-around.

Ok above analysis was wrong. Above should not have been an error message, but rather info. I am just not able to reproduce issue reported at all NOW. I could repro yesterday but I just cant now.

If anyone hits this issue please share the kube-router logs.

[murali-reddy]

So i enabled pprof and gave multiple tries (just delete the kube-router pods and let them re-create), i was able to reproduce the issue. So basically it boils down to the below panic in GoBGP. Its simple addnieghbour call. Irrespective there should not be a panic from the depedent libraries. Will have to see why there is panic in GoBGP api call. Only thing that was added (that is relevent) from 0.0.20 release is locking adding which is logically right thing to do. I will find out why there is crash now, was it there in earlier release also but not surfaced.

1 @ 0x42cb6a 0x42cc4e 0x404e31 0x404a65 0x1347dd7 0x457528 0x42a92f 0x4297de 0x43fe1f 0x131eb69 0x132e118 0x142f2c5 0xf60e4a 0xf4639a 0x459c71
#       0x1347dd6       github.com/cloudnativelabs/kube-router/vendor/github.com/osrg/gobgp/server.(*BgpServer).mgmtOperation.func1+0x56        /home/kube/go/src/github.com/cloudnativelabs/kube-router/vendor/github.com/osrg/gobgp/server/server.go:173
#       0x457527        runtime.call32+0x47                                                                                                     /usr/lib/go-1.8/src/runtime/asm_amd64.s:514
#       0x42a92e        runtime.gopanic+0x2ce                                                                                                   /usr/lib/go-1.8/src/runtime/panic.go:489
#       0x4297dd        runtime.panicmem+0x5d                                                                                                   /usr/lib/go-1.8/src/runtime/panic.go:63
#       0x43fe1e        runtime.sigpanic+0x29e                                                                                                  /usr/lib/go-1.8/src/runtime/signal_unix.go:290
#       0x131eb69       github.com/cloudnativelabs/kube-router/vendor/github.com/osrg/gobgp/server.(*BgpServer).mgmtOperation+0xe9              /home/kube/go/src/github.com/cloudnativelabs/kube-router/vendor/github.com/osrg/gobgp/server/server.go:174
#       0x132e117       github.com/cloudnativelabs/kube-router/vendor/github.com/osrg/gobgp/server.(*BgpServer).AddNeighbor+0x87                /home/kube/go/src/github.com/cloudnativelabs/kube-router/vendor/github.com/osrg/gobgp/server/server.go:1809
#       0x142f2c4       github.com/cloudnativelabs/kube-router/app/controllers.(*NetworkRoutingController).OnNodeUpdate+0x4f4                   /home/kube/go/src/github.com/cloudnativelabs/kube-router/app/controllers/network_routes_controller.go:1226
#       0xf60e49        github.com/cloudnativelabs/kube-router/app/watchers.(*nodeWatcher).RegisterHandler.func1+0x49                           /home/kube/go/src/github.com/cloudnativelabs/kube-router/app/watchers/node_watcher.go:57
#       0xf46399        github.com/cloudnativelabs/kube-router/utils.ListenerFunc.OnUpdate+0x39                                                 /home/kube/go/src/github.com/cloudnativelabs/kube-router/utils/utils.go:14

[roffe]

murali.reddy
no, i figured what happened with last RC1 build

in short below diff fixed it

+       mu                   *sync.Mutex ```

from 
https://github.com/golang/go/issues/20060

we should be able to close this once we got a RC2 out with the pointer to the mutex instead of a copy