CI: K8sServicesTest Checks service across nodes Tests NodePort BPF Tests with direct routing Tests with secondary NodePort device: Exitcode: 42

[aanm]

/home/jenkins/workspace/Cilium-PR-K8s-1.12-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:514
Request from testclient-l9jhj pod to service tftp://10.102.0.52:10069/hello failed
Expected command: kubectl exec -n default testclient-l9jhj -- /bin/bash -c 'fails=""; id=$RANDOM; for i in $(seq 1 10); do if curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 tftp://10.102.0.52:10069/hello -H "User-Agent: cilium-test-$id/$i"; then echo "Test round $id/$i exit code: $?"; else fails=$fails:$id/$i=$?; fi; done; if [ -n "$fails" ]; then echo "failed: $fails"; fi; cnt="${fails//[^:]}"; if [ ${#cnt} -gt 0 ]; then exit 42; fi' 
To succeed, but it failed:
Exitcode: 42 
Err: exit status 42
Stdout:
 	 
	 Hostname: testds-9gdwc
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=49833
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/1 exit code: 0
	 
	 Hostname: testds-4cvff
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=38054
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/2 exit code: 0
	 
	 Hostname: testds-9gdwc
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=50368
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/3 exit code: 0
	 
	 Hostname: testds-9gdwc
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=60718
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/4 exit code: 0
	 
	 Hostname: testds-9gdwc
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=52418
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/5 exit code: 0
	 
	 Hostname: testds-4cvff
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=38251
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/6 exit code: 0
	 
	 Hostname: testds-4cvff
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=34753
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/8 exit code: 0
	 
	 Hostname: testds-9gdwc
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=58050
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/9 exit code: 0
	 
	 Hostname: testds-4cvff
	 
	 Request Information:
	 	client_address=10.0.1.205
	 	client_port=55828
	 	real path=/hello
	 	request_scheme=tftp
	 
	 Test round 26023/10 exit code: 0
	 failed: :26023/7=72
	 
Stderr:
 	 command terminated with exit code 42
	 

/home/jenkins/workspace/Cilium-PR-K8s-1.12-net-next/src/github.com/cilium/cilium/test/k8sT/Services.go:756

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.12-net-next/30/testReport/junit/Suite-k8s-1/12/K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing_Tests_with_secondary_NodePort_device/

fd3841ce_K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing_Tests_with_secondary_NodePort_device.zip

[pchaigno]

I assume variants of this flake with HTTP and VXLAN (e.g., https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19/80/testReport/junit/Suite-k8s-1/20/K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_secondary_NodePort_device/) likely have the same root cause. Or do I need to open a separate issue?

[pchaigno]

@twpayne Should we disable this test?

[twpayne]

@twpayne Should we disable this test?

Yes, given that it's been flaky for 18 months, it's right to disable it.

[brb]

Considering recent K8sServices suite refactoring, the flake might have fixed. Let's reopen if we gonna hit it on master branch again.