CI: K8sServicesTest Checks service across nodes Tests NodePort BPF BPF NAT engine handles unknown protocol packets Should not drop SCTP packets

[jrajahalme]

In https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19/918/testReport/junit/Suite-k8s-1/20/K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_BPF_NAT_engine_handles_unknown_protocol_packets_Should_not_drop_SCTP_packets/:

/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Expected
    <int>: 0
to be >
    <int>: 0
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/Services.go:1393

The problem here is that iperf transmits no data (0 bytes), which is not greater than 0.

Test output (stderr) shows that iperf3 fails with an error:

iperf3: error - control socket has closed unexpectedly
command terminated with exit code 1

iperf pod is in hostnetworking mode, if that makes a difference.

Host endpoint has no policy enforcement, but has not seen any traffic either:

/sys/fs/bpf/tc/globals/cilium_policy_01097:

POLICY   DIRECTION   IDENTITY   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
Allow    Ingress     0          ANY          NONE         0       0         
Allow    Egress      0          ANY          NONE         0       0         

e3cb6a69_K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_BPF_NAT_engine_handles_unknown_protocol_packets_Should_not_drop_SCTP_packets (1).zip

[pchaigno]

Initial Analysis

iperf3-server-9pf5x, the source pod is on k8s2, with IP address 192.168.36.12. It's trying to reach 192.168.36.11 so we don't have any visibility on that traffic (BPF programs are attached on native devices but monitor aggregation is enabled so no {to,from}-network traces).

According to test-output.log, the iperf3 client failed with:

iperf3: error - control socket has closed unexpectedly

The server failed with:

iperf3: error - unable to read from stream socket: Bad file descriptor

Many reports of this error on the iperf repository, so probably fairly generic.

CI Dashboard Analysis

Via the CI Dashboard, I found one other case on master, dating from June 30th:
https://jenkins.cilium.io/job/cilium-master-k8s-1.16-kernel-net-next/549/testReport/junit/Suite-k8s-1/16/K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_BPF_NAT_engine_handles_unknown_protocol_packets_Should_not_drop_SCTP_packets/
20394869_K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_BPF_NAT_engine_handles_unknown_protocol_packets_Should_not_drop_SCTP_packets.zip

Next Steps

We are missing the SCTP ss output to be able to check the listening/established connections. I'll send a PR to fix that. We can also temporarily disable monitor aggregation for that test.