datapath: panic explicitly when IP of direct-routing-device not found #17064

ArthurChiao · 2021-08-06T02:12:59Z

A user reported the follow panic on agent startup:

level=info msg="  --devices=''" subsys=daemon
level=info msg="  --direct-routing-device='wg0'" subsys=daemon
...
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
...
level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
panic: runtime error: index out of range [3] with length 0

goroutine 1 [running]:
encoding/binary.bigEndian.Uint32(...)
        /usr/local/go/src/encoding/binary/binary.go:112
github.com/cilium/cilium/pkg/byteorder.HostSliceToNetwork(0x0, 0x0, 0x0, 0xa, 0x4795aa0, 0x41983f8)
        /go/src/github.com/cilium/cilium/pkg/byteorder/byteorder.go:134 +0x24f
github.com/cilium/cilium/pkg/datapath/linux/config.(*HeaderfileWriter).WriteNodeConfig(0x4792cf0, 0x2e86b00, 0xc0005b4fe8, 0xc0000fcc68, 0x0, 0x0)
        /go/src/github.com/cilium/cilium/pkg/datapath/linux/config/config.go:426 +0x47db
github.com/cilium/cilium/daemon/cmd.(*Daemon).createNodeConfigHeaderfile(0xc0001d5200, 0x0, 0x0)
        /go/src/github.com/cilium/cilium/daemon/cmd/datapath.go:72 +0x3e9
github.com/cilium/cilium/daemon/cmd.(*Daemon).init(0xc0001d5200, 0x4239c20, 0x42278e0)
        /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:233 +0x6a6

With some investigations, this is how the panic happened:

--devices='' && --direct-routing-device='wg0' resulted in final option.Config.Devices=["eth0"] on his machine; which further led to
no ip address was initialized in NodePort IPv4 address map for wg0 device, then
nodePortIPv4Addrs[directRoutingIface] returned an empty net.IP object, and the subsequent byteorder convertion paniced as above

Although it's the user to blame for misconfiguration, the panic message
is not so friendly either for ordinary users determining what's happened
and how to fix it.

This patch improves it by checking the existence of the IP address
before using it, and panic explicitly with more user-friendly messages.

Update: this patch also helps even if c042c05 is added recently, as
the latter also accesses indexes before checking IP existence:

10 // NetIPv4ToHost32 converts an net.IP to a uint32 in host byte order. ip
11 // must be a IPv4 address, otherwise the function will panic.
12 func NetIPv4ToHost32(ip net.IP) uint32 {
13     ipv4 := ip.To4()
14     _ = ipv4[3] // Assert length of ipv4.
15     return Native.Uint32(ipv4)
16 }

Signed-off-by: ArthurChiao arthurchiao@hotmail.com

ti-mo

Thanks! Panicking is always a bug, even if it was caused by the user. :) Left a few comments regarding the dumping of existing addresses on the host.

pkg/datapath/linux/config/config.go

brb · 2021-08-09T14:28:25Z

@ArthurChiao Thanks for the PR!

--devices='' && --direct-routing-device='wg0' resulted in final option.Config.Devices=["eth0"] on his machine; which further led to
no ip address were initialized in NodePort IPv4 address map, then

Are you saying that in their case eth0 didn't have any IP addr assigned?

ArthurChiao · 2021-08-11T14:42:34Z

Are you saying that in their case eth0 didn't have any IP addr assigned?

Well, it had, and the agent log showed that after auto-detecting (--devices=''), it had a result as (i lost the full log user provided):

option.Config.Devices=["eth0"]

Later, the agent tried to get an IP address with key "wg0" (--direct-routing-device='wg0') and the panic happened.

@brb

Update: just noticed that the previous commit message was not accurate enough, maybe that's why you are concerned, I just re-edited it:

With some investigations, this is how the panic happened:

--devices='' && --direct-routing-device='wg0' resulted in final option.Config.Devices=["eth0"] on his machine; which further led to

no ip address was initialized in NodePort IPv4 address map for wg0 device, then

nodePortIPv4Addrs["wg0"] returned an empty net.IP object, and the subsequent byteorder convertion paniced as above

ti-mo · 2021-08-12T11:16:40Z

test-me-please

ti-mo · 2021-08-16T09:39:25Z

@ArthurChiao Could you give this a rebase please? The tests failing here were fixed on master. Sorry for the back and forth.

A user reported the follow panic on agent startup: ``` level=info msg=" --devices=''" subsys=daemon level=info msg=" --direct-routing-device='wg0'" subsys=daemon ... level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon ... level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon panic: runtime error: index out of range [3] with length 0 goroutine 1 [running]: encoding/binary.bigEndian.Uint32(...) /usr/local/go/src/encoding/binary/binary.go:112 github.com/cilium/cilium/pkg/byteorder.HostSliceToNetwork(0x0, 0x0, 0x0, 0xa, 0x4795aa0, 0x41983f8) /go/src/github.com/cilium/cilium/pkg/byteorder/byteorder.go:134 +0x24f github.com/cilium/cilium/pkg/datapath/linux/config.(*HeaderfileWriter).WriteNodeConfig(0x4792cf0, 0x2e86b00, 0xc0005b4fe8, 0xc0000fcc68, 0x0, 0x0) /go/src/github.com/cilium/cilium/pkg/datapath/linux/config/config.go:426 +0x47db github.com/cilium/cilium/daemon/cmd.(*Daemon).createNodeConfigHeaderfile(0xc0001d5200, 0x0, 0x0) /go/src/github.com/cilium/cilium/daemon/cmd/datapath.go:72 +0x3e9 github.com/cilium/cilium/daemon/cmd.(*Daemon).init(0xc0001d5200, 0x4239c20, 0x42278e0) /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:233 +0x6a6 ``` With some investigations, this is how the panic happened: 1. --devices='' && --direct-routing-device='wg0' resulted in final option.Config.Devices=["eth0"] on his machine; which further led to 2. no ip address were initialized in NodePort IPv4 address map for `wg0`, then 3. `nodePortIPv4Addrs["wg0"]` returned an empty net.IP object, and the subsequent byteorder convertion paniced as above Although it's the user to blame for misconfiguration, the panic message is not so friendly either for ordinary users determining what's happened and how to fix it. This patch improves it by checking the existance of the IP address before using it, and panic explicitly with more user-friendly messages. Update: this patch also helps even if c042c05 is added recently, as the latter also accesses indexes before checking IP existance: ```go 10 // NetIPv4ToHost32 converts an net.IP to a uint32 in host byte order. ip 11 // must be a IPv4 address, otherwise the function will panic. 12 func NetIPv4ToHost32(ip net.IP) uint32 { 13 ipv4 := ip.To4() 14 _ = ipv4[3] // Assert length of ipv4. 15 return Native.Uint32(ipv4) 16 } ``` Signed-off-by: ArthurChiao <arthurchiao@hotmail.com>

ArthurChiao · 2021-08-16T13:25:01Z

Hi @ti-mo, that's all right, just rebased and pushed, let's see if it could pass all the tests. Thanks for your time!

ti-mo · 2021-08-16T16:33:27Z

test-me-please

Job 'Cilium-PR-K8s-1.16-net-next' failed and has not been observed before, so may be related to your PR:

Click to show.

Test Name

K8sServicesTest Checks service across nodes Tests NodePort BPF Tests with direct routing Test NodePort with netfilterCompatMode=true

Failure Output

FAIL: Request from testclient-rfgbn pod to service tftp://192.168.36.12:32254/hello failed

If it is a flake, comment /mlh new-flake Cilium-PR-K8s-1.16-net-next so I can create a new GitHub issue to track it.

ArthurChiao · 2021-08-17T03:21:25Z

Hi @ti-mo, I looked into the tests logs, here are some findings:

The test failed when accessing testds service from testclient (noticed that testds-rpd6d started 8m30s after testds-t4m9h and all the clients, but it should be OK):

	 default             testclient-rfgbn                   1/1     Running   0          12m     10.0.0.59       k8s2   <none>           <none>
	 default             testclient-s578d                   1/1     Running   0          12m     10.0.1.196      k8s1   <none>           <none>
	 default             testds-rpd6d                       2/2     Running   0          2m26s   10.0.1.135      k8s1   <none>           <none>
	 default             testds-t4m9h                       2/2     Running   0          11m     10.0.0.84       k8s2   <none>           <none

The cilium service list showed that requests would be load balanced to the two instances shown above:

	 13   10.0.2.15:32254         NodePort       1 => 10.0.1.135:69        
	                                             2 => 10.0.0.84:69         
	 14   192.168.36.12:32254     NodePort       1 => 10.0.1.135:69        
	                                             2 => 10.0.0.84:69         
	 15   0.0.0.0:32254           NodePort       1 => 10.0.1.135:69        
	                                             2 => 10.0.0.84:69

Searching the failed attempts with src_port info from the accessing logs of the server side, it seemed that all requests have been received and replied:

$ grep 10.0.0.59 -n pod-default-testds-rpd6d-udp.log
2021-08-16T19:39:19.527357517Z 2021/08/16 19:39:19 Listening on: ":69"
	Line 11: 2021-08-16T19:40:34.542192339Z 2021/08/16 19:40:34 Read "hello" requested from 10.0.0.59:35673
	Line 12: 2021-08-16T19:40:34.551000536Z 2021/08/16 19:40:34 Read "hello" requested from 10.0.0.59:57298
	Line 13: 2021-08-16T19:40:34.555028778Z 2021/08/16 19:40:34 Read "hello" requested from 10.0.0.59:40567
	Line 14: 2021-08-16T19:40:34.564862416Z 2021/08/16 19:40:34 Read "hello" requested from 10.0.0.59:46367
	Line 27: 2021-08-16T19:41:19.750385895Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:40700
	Line 28: 2021-08-16T19:41:19.766217533Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:60624
	Line 29: 2021-08-16T19:41:19.781887486Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:45186
	Line 30: 2021-08-16T19:41:19.809568617Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:43653
	Line 31: 2021-08-16T19:41:19.822328471Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:60715
	Line 32: 2021-08-16T19:41:19.911977073Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:47404
	Line 33: 2021-08-16T19:41:19.957047857Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:40505
	Line 34: 2021-08-16T19:41:19.995829848Z 2021/08/16 19:41:19 Read "hello" requested from 10.0.0.59:38980
	Line 38: 2021-08-16T19:41:20.231881115Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:34572
	Line 40: 2021-08-16T19:41:20.294854477Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:58174
	Line 42: 2021-08-16T19:41:20.348616936Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:36617
	Line 43: 2021-08-16T19:41:20.441396284Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:40741
	Line 44: 2021-08-16T19:41:20.558574897Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:38103
	Line 45: 2021-08-16T19:41:20.594025009Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:56268
	Line 47: 2021-08-16T19:41:20.627062402Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:45123
	Line 49: 2021-08-16T19:41:20.682662873Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:48779
	Line 51: 2021-08-16T19:41:20.695741554Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:51739
	Line 52: 2021-08-16T19:41:20.696773660Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:38224
	Line 53: 2021-08-16T19:41:20.726321515Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:40474
	Line 54: 2021-08-16T19:41:20.737045513Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:43766
	Line 55: 2021-08-16T19:41:20.745269555Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:51858
	Line 57: 2021-08-16T19:41:20.780929209Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:39420
	Line 59: 2021-08-16T19:41:20.843478954Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:46367
	Line 63: 2021-08-16T19:41:20.887740695Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:58965
...

$ grep 10.0.0.59 -n pod-default-testds-t4m9h-udp.log
2021-08-16T19:30:24.948438313Z 2021/08/16 19:30:24 Listening on: ":69"
Line 17: 2021-08-16T19:32:59.346324828Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:54063
	Line 20: 2021-08-16T19:32:59.373127283Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:45664
	Line 21: 2021-08-16T19:32:59.422351574Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:56371
	Line 24: 2021-08-16T19:32:59.505363326Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:57805
	Line 25: 2021-08-16T19:32:59.578068246Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:45649
	Line 26: 2021-08-16T19:32:59.622611823Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:39984
	Line 28: 2021-08-16T19:32:59.654513490Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:35904
	Line 29: 2021-08-16T19:32:59.666853778Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:34387
	Line 32: 2021-08-16T19:32:59.738101732Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:36831
	Line 33: 2021-08-16T19:32:59.773680027Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:34358
	Line 35: 2021-08-16T19:32:59.826318540Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:43552
	Line 37: 2021-08-16T19:32:59.836200289Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:55421
	Line 38: 2021-08-16T19:32:59.843909642Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:45621
	Line 39: 2021-08-16T19:32:59.858143783Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:57819
	Line 40: 2021-08-16T19:32:59.861814655Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:37869
	Line 42: 2021-08-16T19:32:59.874264841Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:33245
	Line 43: 2021-08-16T19:32:59.882402694Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:56993
	Line 44: 2021-08-16T19:32:59.899424928Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:48586
	Line 46: 2021-08-16T19:32:59.911807822Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:39837
	Line 48: 2021-08-16T19:32:59.967878884Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:33377
	Line 51: 2021-08-16T19:33:00.000558598Z 2021/08/16 19:32:59 Read "hello" requested from 10.0.0.59:53180
	Line 52: 2021-08-16T19:33:00.038334403Z 2021/08/16 19:33:00 Read "hello" requested from 10.0.0.59:57274
	Line 55: 2021-08-16T19:33:00.131305877Z 2021/08/16 19:33:00 Read "hello" requested from 10.0.0.59:52165
	Line 57: 2021-08-16T19:33:00.164746573Z 2021/08/16 19:33:00 Read "hello" requested from 10.0.0.59:45535
	Line 120: 2021-08-16T19:34:22.407315090Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:34771
	Line 121: 2021-08-16T19:34:22.420839256Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:41730
	Line 122: 2021-08-16T19:34:22.435372586Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:47402
	Line 123: 2021-08-16T19:34:22.449974424Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:57516
	Line 124: 2021-08-16T19:34:22.485655943Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:48785
	Line 130: 2021-08-16T19:34:22.712823273Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:60268
	Line 132: 2021-08-16T19:34:22.731360151Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:55464
	Line 133: 2021-08-16T19:34:22.738619389Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:51275
	Line 135: 2021-08-16T19:34:22.768591845Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:48078
	Line 137: 2021-08-16T19:34:22.786527330Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:56795
	Line 138: 2021-08-16T19:34:22.795399782Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:40106
	Line 144: 2021-08-16T19:34:22.930664627Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:36922
	Line 145: 2021-08-16T19:34:22.942872836Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:53992
	Line 150: 2021-08-16T19:34:22.995311806Z 2021/08/16 19:34:22 Read "hello" requested from 10.0.0.59:44909
	Line 151: 2021-08-16T19:34:23.008958035Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:37432
	Line 153: 2021-08-16T19:34:23.014645445Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:51565
	Line 155: 2021-08-16T19:34:23.028404837Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:33047
	Line 156: 2021-08-16T19:34:23.030242932Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:46613
	Line 157: 2021-08-16T19:34:23.039965231Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:51149
	Line 158: 2021-08-16T19:34:23.060088365Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:58496
	Line 159: 2021-08-16T19:34:23.093075888Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:35984
	Line 160: 2021-08-16T19:34:23.098949720Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:56491
	Line 161: 2021-08-16T19:34:23.213070844Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:57323
	Line 162: 2021-08-16T19:34:23.238965790Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:50565
	Line 163: 2021-08-16T19:34:23.268090316Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:60065
	Line 164: 2021-08-16T19:34:23.313473028Z 2021/08/16 19:34:23 Read "hello" requested from 10.0.0.59:45094
	Line 207: 2021-08-16T19:35:36.328019718Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:57944
	Line 208: 2021-08-16T19:35:36.346698130Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:38373
	Line 209: 2021-08-16T19:35:36.364116360Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:42021
	Line 210: 2021-08-16T19:35:36.409222203Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:46275
	Line 211: 2021-08-16T19:35:36.409340341Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:55086
	Line 212: 2021-08-16T19:35:36.419359413Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:33297
	Line 213: 2021-08-16T19:35:36.504864263Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:56871
	Line 214: 2021-08-16T19:35:36.510190452Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:51818
	Line 215: 2021-08-16T19:35:36.540156799Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:43828
	Line 217: 2021-08-16T19:35:36.578930339Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:44458
	Line 219: 2021-08-16T19:35:36.606494998Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:52303
	Line 220: 2021-08-16T19:35:36.625380425Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:54729
	Line 223: 2021-08-16T19:35:36.661930014Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:48925
	Line 225: 2021-08-16T19:35:36.712311450Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:57117
	Line 226: 2021-08-16T19:35:36.745914049Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:37677
	Line 228: 2021-08-16T19:35:36.763122439Z 2021/08/16 19:35:36 Read "hello" requested from 10.0.0.59:57864
	Line 234: 2021-08-16T19:35:37.111208554Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:48179
	Line 237: 2021-08-16T19:35:37.139235362Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:44302
	Line 240: 2021-08-16T19:35:37.173087024Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:54415
	Line 241: 2021-08-16T19:35:37.195307704Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:56015
	Line 243: 2021-08-16T19:35:37.228538687Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:34204
	Line 250: 2021-08-16T19:35:37.436292322Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:58887
	Line 252: 2021-08-16T19:35:37.485080431Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:58365
	Line 254: 2021-08-16T19:35:37.511140514Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:45100
	Line 256: 2021-08-16T19:35:37.527753671Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:57075
	Line 257: 2021-08-16T19:35:37.640968401Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:57175
	Line 258: 2021-08-16T19:35:37.665080975Z 2021/08/16 19:35:37 Read "hello" requested from 10.0.0.59:49114
	Line 304: 2021-08-16T19:37:12.869994145Z 2021/08/16 19:37:12 Read "hello" requested from 10.0.0.59:49244
	Line 305: 2021-08-16T19:37:12.912844771Z 2021/08/16 19:37:12 Read "hello" requested from 10.0.0.59:40239
	Line 306: 2021-08-16T19:37:12.919328559Z 2021/08/16 19:37:12 Read "hello" requested from 10.0.0.59:38981
	Line 307: 2021-08-16T19:37:12.996937314Z 2021/08/16 19:37:12 Read "hello" requested from 10.0.0.59:46089
	Line 308: 2021-08-16T19:37:13.007970059Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:58496
	Line 309: 2021-08-16T19:37:13.013918495Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:42910
	Line 310: 2021-08-16T19:37:13.086494640Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:43097
	Line 313: 2021-08-16T19:37:13.267164976Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:59306
	Line 316: 2021-08-16T19:37:13.465084948Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:58792
	Line 317: 2021-08-16T19:37:13.507410978Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:55761
	Line 318: 2021-08-16T19:37:13.558960266Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:36862
	Line 319: 2021-08-16T19:37:13.574007126Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:47704
	Line 321: 2021-08-16T19:37:13.680368514Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:41913
	Line 322: 2021-08-16T19:37:13.718338627Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:57925
	Line 323: 2021-08-16T19:37:13.726282511Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:53039
	Line 325: 2021-08-16T19:37:13.775822135Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:51300
	Line 328: 2021-08-16T19:37:13.804021706Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:53760
	Line 329: 2021-08-16T19:37:13.812799533Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:57778
	Line 330: 2021-08-16T19:37:13.820130482Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:44184
	Line 332: 2021-08-16T19:37:13.827156656Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:54347
	Line 334: 2021-08-16T19:37:13.840145183Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:50137
	Line 336: 2021-08-16T19:37:13.854465934Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:37785
	Line 338: 2021-08-16T19:37:13.902629573Z 2021/08/16 19:37:13 Read "hello" requested from 10.0.0.59:47075
	Line 418: 2021-08-16T19:38:24.453373733Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:55321
	Line 419: 2021-08-16T19:38:24.510258493Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:48139
	Line 420: 2021-08-16T19:38:24.539801822Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:37970
	Line 421: 2021-08-16T19:38:24.567485143Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:58237
	Line 422: 2021-08-16T19:38:24.594377199Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:40574
	Line 423: 2021-08-16T19:38:24.616304714Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:37398
	Line 424: 2021-08-16T19:38:24.644074988Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:47753
	Line 425: 2021-08-16T19:38:24.687186790Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:41427
	Line 426: 2021-08-16T19:38:24.706925464Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:53370
	Line 427: 2021-08-16T19:38:24.750509713Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:42894
	Line 428: 2021-08-16T19:38:24.750720152Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:39145
	Line 429: 2021-08-16T19:38:24.796955107Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:44454
	Line 430: 2021-08-16T19:38:24.839627740Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:55059
	Line 431: 2021-08-16T19:38:24.850269859Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:53937
	Line 433: 2021-08-16T19:38:24.881472920Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:50837
	Line 434: 2021-08-16T19:38:24.904485934Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:55811
	Line 435: 2021-08-16T19:38:24.905488771Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:46477
	Line 436: 2021-08-16T19:38:24.918605486Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:38456
	Line 437: 2021-08-16T19:38:24.924302345Z 2021/08/16 19:38:24 Read "hello" requested from 10.0.0.59:60238
	Line 441: 2021-08-16T19:38:25.370650540Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:41976
	Line 442: 2021-08-16T19:38:25.389630766Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:49460
	Line 443: 2021-08-16T19:38:25.401332495Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:59900
	Line 444: 2021-08-16T19:38:25.413791699Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:60549
	Line 445: 2021-08-16T19:38:25.437051592Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:46974
	Line 447: 2021-08-16T19:38:25.518677560Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:50443
	Line 448: 2021-08-16T19:38:25.543571416Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:44279
	Line 449: 2021-08-16T19:38:25.556522666Z 2021/08/16 19:38:25 Read "hello" requested from 10.0.0.59:49369
	Line 523: 2021-08-16T19:41:20.499706276Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:40665
	Line 524: 2021-08-16T19:41:20.510543679Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:57350
	Line 526: 2021-08-16T19:41:20.518021164Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:51894
	Line 528: 2021-08-16T19:41:20.561911514Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:43056
	Line 529: 2021-08-16T19:41:20.602856072Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:46118
	Line 530: 2021-08-16T19:41:20.644281038Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:51729
	Line 531: 2021-08-16T19:41:20.644317492Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:58855
	Line 532: 2021-08-16T19:41:20.684198784Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:56433
	Line 533: 2021-08-16T19:41:20.727260505Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:38213
	Line 534: 2021-08-16T19:41:20.772902817Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:39291
	Line 535: 2021-08-16T19:41:20.790627425Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:51686
	Line 536: 2021-08-16T19:41:20.805105367Z 2021/08/16 19:41:20 Read "hello" requested from 10.0.0.59:40062
	Line 539: 2021-08-16T19:41:21.045634667Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:40734
	Line 540: 2021-08-16T19:41:21.059369404Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:50072
	Line 541: 2021-08-16T19:41:21.067159421Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:48290
	Line 543: 2021-08-16T19:41:21.126557878Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:36385
	Line 545: 2021-08-16T19:41:21.165164940Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:39639
	Line 547: 2021-08-16T19:41:21.208356246Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:54024
	Line 549: 2021-08-16T19:41:21.433872227Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:50692
	Line 550: 2021-08-16T19:41:21.476393720Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:33424
	Line 551: 2021-08-16T19:41:21.502154129Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:56838
	Line 552: 2021-08-16T19:41:21.517978078Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:59287
	Line 554: 2021-08-16T19:41:21.532835518Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:57106
	Line 555: 2021-08-16T19:41:21.551791205Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:56011
	Line 556: 2021-08-16T19:41:21.572235788Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:55455
	Line 558: 2021-08-16T19:41:21.714051464Z 2021/08/16 19:41:21 Read "hello" requested from 10.0.0.59:42762

So I'm guessing is there asymmetric path problem in the test setup? (I have looked into the routing table in the debuginfo, but havn't find anything useful, as I'm less familir with the networking mode in public cloud) After all, I can see why adding several lines of logs would break the connectivity. Thanks!

ti-mo · 2021-08-17T09:03:43Z

@ArthurChiao There are some issues with our tests, I will rebase and merge when they are fixed. This change should not impact any cluster connectivity as you've mentioned.

ti-mo · 2021-08-17T09:04:30Z

test-me-please

Job 'Cilium-PR-K8s-1.16-net-next' has 1 failure but they might be new flake since it also hit 1 known flake: #17060 (84.77)

ti-mo · 2021-08-17T13:39:36Z

Hit by known net-next flake, marking as ready.

ArthurChiao requested a review from a team as a code owner August 6, 2021 02:13

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Aug 6, 2021

ArthurChiao requested a review from ti-mo August 6, 2021 02:13

maintainer-s-little-helper bot assigned ti-mo Aug 6, 2021

aanm requested review from a team and joamaki and removed request for a team August 6, 2021 20:55

maintainer-s-little-helper bot assigned joamaki Aug 6, 2021

aanm removed the request for review from joamaki August 6, 2021 20:55

aanm unassigned joamaki Aug 6, 2021

ArthurChiao force-pushed the check_direct_routing_ip_existance_before_unexpected_panic branch from 88cbfe7 to e873f07 Compare August 9, 2021 01:21

ti-mo requested changes Aug 9, 2021

View reviewed changes

pkg/datapath/linux/config/config.go Outdated Show resolved Hide resolved

pkg/datapath/linux/config/config.go Outdated Show resolved Hide resolved

pkg/datapath/linux/config/config.go Outdated Show resolved Hide resolved

pkg/datapath/linux/config/config.go Outdated Show resolved Hide resolved

ArthurChiao force-pushed the check_direct_routing_ip_existance_before_unexpected_panic branch 4 times, most recently from 97c80e4 to 0cbe943 Compare August 12, 2021 02:07

ArthurChiao requested a review from ti-mo August 12, 2021 10:33

ti-mo approved these changes Aug 12, 2021

View reviewed changes

maintainer-s-little-helper bot unassigned ti-mo Aug 12, 2021

ti-mo added area/daemon Impacts operation of the Cilium daemon. release-note/bug This PR fixes an issue in a previous release of Cilium. labels Aug 12, 2021

maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Aug 12, 2021

ti-mo added the needs-backport/1.10 label Aug 12, 2021

maintainer-s-little-helper bot added this to Needs backport from master in 1.10.4 Aug 12, 2021

ArthurChiao force-pushed the check_direct_routing_ip_existance_before_unexpected_panic branch from 0cbe943 to 76f91aa Compare August 16, 2021 13:23

ti-mo self-assigned this Aug 17, 2021

maintainer-s-little-helper bot mentioned this pull request Aug 17, 2021

CI: K8sServicesTest Checks service across nodes Tests NodePort BPF Tests with vxlan Test NodePort with netfilterCompatMode=true #17060

Closed

ti-mo added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Aug 17, 2021

ti-mo merged commit 83edea8 into cilium:master Aug 17, 2021

ArthurChiao deleted the check_direct_routing_ip_existance_before_unexpected_panic branch August 17, 2021 13:44

ti-mo mentioned this pull request Aug 18, 2021

v1.10 backports 2021-08-18 #17183

Merged

ti-mo added backport-pending/1.10 and removed needs-backport/1.10 labels Aug 18, 2021

maintainer-s-little-helper bot moved this from Needs backport from master to Backport pending to v1.10 in 1.10.4 Aug 18, 2021

tklauser added backport-done/1.10 and removed backport-pending/1.10 labels Aug 19, 2021

maintainer-s-little-helper bot moved this from Backport pending to v1.10 to Backport done to v1.10 in 1.10.4 Aug 19, 2021

joestringer mentioned this pull request Sep 1, 2021

Prepare for release v1.10.4 #17287

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

datapath: panic explicitly when IP of direct-routing-device not found #17064

datapath: panic explicitly when IP of direct-routing-device not found #17064

ArthurChiao commented Aug 6, 2021 •

edited

ti-mo left a comment

brb commented Aug 9, 2021

ArthurChiao commented Aug 11, 2021 •

edited

ti-mo commented Aug 12, 2021

ti-mo commented Aug 16, 2021

ArthurChiao commented Aug 16, 2021

ti-mo commented Aug 16, 2021 •

edited by maintainer-s-little-helper bot

Test Name

Failure Output

ArthurChiao commented Aug 17, 2021

ti-mo commented Aug 17, 2021

ti-mo commented Aug 17, 2021 •

edited by maintainer-s-little-helper bot

ti-mo commented Aug 17, 2021

datapath: panic explicitly when IP of direct-routing-device not found #17064

datapath: panic explicitly when IP of direct-routing-device not found #17064

Conversation

ArthurChiao commented Aug 6, 2021 • edited

ti-mo left a comment

Choose a reason for hiding this comment

brb commented Aug 9, 2021

ArthurChiao commented Aug 11, 2021 • edited

ti-mo commented Aug 12, 2021

ti-mo commented Aug 16, 2021

ArthurChiao commented Aug 16, 2021

ti-mo commented Aug 16, 2021 • edited by maintainer-s-little-helper bot

Test Name

Failure Output

ArthurChiao commented Aug 17, 2021

ti-mo commented Aug 17, 2021

ti-mo commented Aug 17, 2021 • edited by maintainer-s-little-helper bot

ti-mo commented Aug 17, 2021

ArthurChiao commented Aug 6, 2021 •

edited

ArthurChiao commented Aug 11, 2021 •

edited

ti-mo commented Aug 16, 2021 •

edited by maintainer-s-little-helper bot

ti-mo commented Aug 17, 2021 •

edited by maintainer-s-little-helper bot