CI: kubernetes-e2e-net-conformance (ipv4) - NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client #26492

qmonnet · 2023-06-26T23:14:05Z

CI failure

Link
PR (for main branch)
No artifacts

 • [FAILED] [567.806 seconds]
[sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [It] should enforce policy based on PodSelector with MatchExpressions[Feature:NetworkPolicy]
test/e2e/network/netpol/network_legacy.go:270

[...]

  Jun 26 21:28:53.710: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:33299/ --kubeconfig=/home/runner/work/cilium/cilium/_artifacts/kubeconfig.conf --namespace=network-policy-9493 logs client-a-t8xw7 --tail=100'
  Jun 26 21:28:54.425: INFO: stderr: ""
  Jun 26 21:28:54.425: INFO: stdout: "OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted\nOTHER: dial tcp 10.96.174.86:80: connect: operation not permitted\nOTHER: dial tcp 10.96.174.86:80: connect: operation not permitted\nOTHER: dial tcp 10.96.174.86:80: connect: operation not permitted\nOTHER: dial tcp 10.96.174.86:80: connect: operation not permitted\n"
  Jun 26 21:28:54.425: INFO:
  Last 100 log lines of client-a-t8xw7:
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted

[...]

  [FAILED] Pod client-a-t8xw7 should be able to connect to service svc-server, but was not able to connect.
  Pod logs:
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted
  OTHER: dial tcp 10.96.174.86:80: connect: operation not permitted


   Current NetworkPolicies:
  	[{{ } {allow-client-a-via-pod-selector-with-match-expressions  network-policy-9493  2ef78995-2adb-4d1a-b47f-680d7f12ab58 4884 1 2023-06-26 21:22:27 +0000 UTC <nil> <nil> map[] map[] [] [] [{e2e.test Update networking.k8s.io/v1 2023-06-26 21:22:27 +0000 UTC FieldsV1 {"f:spec":{"f:ingress":{},"f:podSelector":{},"f:policyTypes":{}}} }]} {{map[pod-name:server] []} [{[] [{&LabelSelector{MatchLabels:map[string]string{},MatchExpressions:[]LabelSelectorRequirement{LabelSelectorRequirement{Key:pod-name,Operator:In,Values:[client-a],},},} nil nil}]}] [] [Ingress]} {[]}}]

   Pods:
  	[Pod: client-a-t8xw7, Status: &PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:22:27 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:23:34 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:23:34 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:22:27 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.229,StartTime:2023-06-26 21:22:27 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:client,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2023-06-26 21:23:13 +0000 UTC,FinishedAt:2023-06-26 21:23:28 +0000 UTC,ContainerID:containerd://501b494b54394f568902a60f48548197856c65d477623d15863ddb2acff07cd1,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://501b494b54394f568902a60f48548197856c65d477623d15863ddb2acff07cd1,Started:*false,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.229,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
   Pod: server-mrrgw, Status: &PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:19:44 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:28:50 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:28:50 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:19:44 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.110,StartTime:2023-06-26 21:19:44 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:server-container-80,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:20:04 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://713781a6ea78e5cc9a867e3a4ea6fda8e468a71d38f01e41b26c26b9f707fe5f,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},ContainerStatus{Name:server-container-81,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:20:05 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://531731065ce48f4fe9fd20c90f237cf651b67a51b5a4caf05566701ae35315d3,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.110,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
  ]

  In [It] at: test/e2e/network/netpol/network_legacy.go:1944 @ 06/26/23 21:28:54.834

[...]

• [FAILED] [455.502 seconds]
[sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [BeforeEach] should enforce multiple ingress policies with ingress allow-all policy taking precedence [Feature:NetworkPolicy]
  [BeforeEach] test/e2e/network/netpol/network_legacy.go:78
  [It] test/e2e/network/netpol/network_legacy.go:1124

[...]

  Jun 26 21:35:22.624: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:33299/ --kubeconfig=/home/runner/work/cilium/cilium/_artifacts/kubeconfig.conf --namespace=network-policy-9636 logs client-can-connect-80-h49f2 --tail=100'
  Jun 26 21:35:22.886: INFO: stderr: ""
  Jun 26 21:35:22.886: INFO: stdout: "OTHER: dial tcp 10.96.67.231:80: connect: operation not permitted\nOTHER: dial tcp 10.96.67.231:80: connect: operation not permitted\nOTHER: dial tcp 10.96.67.231:80: connect: operation not permitted\nOTHER: dial tcp 10.96.67.231:80: connect: operation not permitted\nOTHER: dial tcp 10.96.67.231:80: connect: operation not permitted\n"

[...]

  [FAILED] Pod client-can-connect-80-h49f2 should be able to connect to service svc-server, but was not able to connect.
  Pod logs:
  OTHER: dial tcp 10.96.67.231:80: connect: operation not permitted
  OTHER: dial tcp 10.96.67.231:80: connect: operation not permitted
  OTHER: dial tcp 10.96.67.231:80: connect: operation not permitted
  OTHER: dial tcp 10.96.67.231:80: connect: operation not permitted
  OTHER: dial tcp 10.96.67.231:80: connect: operation not permitted


   Current NetworkPolicies:
  	[]

   Pods:
  	[Pod: client-can-connect-80-h49f2, Status: &PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:29:28 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:30:14 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:30:14 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:29:28 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.156,StartTime:2023-06-26 21:29:28 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:client,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2023-06-26 21:29:57 +0000 UTC,FinishedAt:2023-06-26 21:30:09 +0000 UTC,ContainerID:containerd://bb1a4457b7856d64359a6be5e2f68ceb3742590cb4b12eee6e1eff9fa7373ce2,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://bb1a4457b7856d64359a6be5e2f68ceb3742590cb4b12eee6e1eff9fa7373ce2,Started:*false,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.156,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
   Pod: server-ps6n4, Status: &PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:28:00 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:32:40 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:32:40 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:28:00 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.145,StartTime:2023-06-26 21:28:00 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:server-container-80,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:28:32 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://b768adb731fba7a914a7238d017a5d0011e3e0a50f565118183643c16c6ea19e,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},ContainerStatus{Name:server-container-81,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:28:44 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://70c72f055299d3fe07fbee48ee59dea0ef34f6ee2e0ad65f2b585f120bbbadfe,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.145,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
  ]

  In [BeforeEach] at: test/e2e/network/netpol/network_legacy.go:1944 @ 06/26/23 21:35:23.441

[...]

• [FAILED] [600.932 seconds]
[sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [It] should enforce policy to allow traffic from pods within server namespace based on PodSelector [Feature:NetworkPolicy]
test/e2e/network/netpol/network_legacy.go:165

[...]

  Jun 26 21:35:07.683: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:33299/ --kubeconfig=/home/runner/work/cilium/cilium/_artifacts/kubeconfig.conf --namespace=network-policy-414 logs client-a-g9cpl --tail=100'
  Jun 26 21:35:07.942: INFO: stderr: ""
  Jun 26 21:35:07.942: INFO: stdout: "OTHER: dial tcp 10.96.76.78:80: connect: operation not permitted\nOTHER: dial tcp 10.96.76.78:80: connect: operation not permitted\nOTHER: dial tcp 10.96.76.78:80: connect: operation not permitted\nOTHER: dial tcp 10.96.76.78:80: connect: operation not permitted\nOTHER: dial tcp 10.96.76.78:80: connect: operation not permitted\n"

[...]

  [FAILED] Pod client-a-g9cpl should be able to connect to service svc-server, but was not able to connect.
  Pod logs:
  OTHER: dial tcp 10.96.76.78:80: connect: operation not permitted
  OTHER: dial tcp 10.96.76.78:80: connect: operation not permitted
  OTHER: dial tcp 10.96.76.78:80: connect: operation not permitted
  OTHER: dial tcp 10.96.76.78:80: connect: operation not permitted
  OTHER: dial tcp 10.96.76.78:80: connect: operation not permitted


   Current NetworkPolicies:
  	[]

   Pods:
  	[Pod: client-a-g9cpl, Status: &PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:28:42 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:29:56 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:29:56 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:28:42 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.83,StartTime:2023-06-26 21:28:42 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:client,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2023-06-26 21:29:40 +0000 UTC,FinishedAt:2023-06-26 21:29:49 +0000 UTC,ContainerID:containerd://07f8236cd4178b6e69daec11498e45ffb4df2567dd84752aa8cda77e91eb0510,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://07f8236cd4178b6e69daec11498e45ffb4df2567dd84752aa8cda77e91eb0510,Started:*false,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.83,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
   Pod: server-mgl8c, Status: &PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:25:40 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:33:27 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:33:27 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:25:40 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.146,StartTime:2023-06-26 21:25:40 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:server-container-80,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:26:17 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://e0f945732e286d307f2a4f3ed9e3e5735043cab2ce18cb24dc7f633a30fffad2,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},ContainerStatus{Name:server-container-81,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:26:31 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://099b71f4323bc6ac5e7e7f5f76f3b4094f0b9b2892821e307d95583d4b638254,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.146,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
  ]

  In [It] at: test/e2e/network/netpol/network_legacy.go:1944 @ 06/26/23 21:35:08.751

[...]

 • [FAILED] [425.045 seconds]
[sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [BeforeEach] should support a 'default-deny-ingress' policy [Feature:NetworkPolicy]
  [BeforeEach] test/e2e/network/netpol/network_legacy.go:78
  [It] test/e2e/network/netpol/network_legacy.go:100

[...]

  Jun 26 21:38:44.378: INFO: Running '/usr/local/bin/kubectl --server=https://127.0.0.1:33299/ --kubeconfig=/home/runner/work/cilium/cilium/_artifacts/kubeconfig.conf --namespace=network-policy-9450 logs client-can-connect-81-rkzvr --tail=100'
  Jun 26 21:38:44.558: INFO: stderr: ""
  Jun 26 21:38:44.558: INFO: stdout: "OTHER: dial tcp 10.96.245.118:81: connect: operation not permitted\nOTHER: dial tcp 10.96.245.118:81: connect: operation not permitted\nOTHER: dial tcp 10.96.245.118:81: connect: operation not permitted\nOTHER: dial tcp 10.96.245.118:81: connect: operation not permitted\nOTHER: dial tcp 10.96.245.118:81: connect: operation not permitted\n"

[...]

  [FAILED] Pod client-can-connect-81-rkzvr should be able to connect to service svc-server, but was not able to connect.
  Pod logs:
  OTHER: dial tcp 10.96.245.118:81: connect: operation not permitted
  OTHER: dial tcp 10.96.245.118:81: connect: operation not permitted
  OTHER: dial tcp 10.96.245.118:81: connect: operation not permitted
  OTHER: dial tcp 10.96.245.118:81: connect: operation not permitted
  OTHER: dial tcp 10.96.245.118:81: connect: operation not permitted


   Current NetworkPolicies:
  	[]

   Pods:
  	[Pod: client-can-connect-81-rkzvr, Status: &PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:32:59 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:33:35 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:33:35 +0000 UTC,Reason:PodFailed,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:32:59 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.228,StartTime:2023-06-26 21:32:59 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:client,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2023-06-26 21:33:20 +0000 UTC,FinishedAt:2023-06-26 21:33:29 +0000 UTC,ContainerID:containerd://dd9b177ffce65e3e351474bde116a3d2d8c140546c62ccc6735ea4a795875da0,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://dd9b177ffce65e3e351474bde116a3d2d8c140546c62ccc6735ea4a795875da0,Started:*false,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.228,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
   Pod: server-5dw4b, Status: &PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:31:46 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:35:12 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:35:12 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-06-26 21:31:46 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:172.18.0.4,PodIP:10.244.1.197,StartTime:2023-06-26 21:31:46 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:server-container-80,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:32:08 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://93807515023797eb507ef0344eaeacc25f519b2ca568e1496b74bfc568efb75d,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},ContainerStatus{Name:server-container-81,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2023-06-26 21:32:12 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:registry.k8s.io/e2e-test-images/agnhost:2.43,ImageID:registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e,ContainerID:containerd://af75bc8489044fbf7bde32377f95adf9330bb9cac28c875b9abe09f9ef503937,Started:*true,AllocatedResources:ResourceList{},Resources:nil,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.1.197,},},EphemeralContainerStatuses:[]ContainerStatus{},Resize:,}
  ]

  In [BeforeEach] at: test/e2e/network/netpol/network_legacy.go:1944 @ 06/26/23 21:38:45.17

[...]

   In [BeforeEach] at: test/e2e/network/netpol/network_legacy.go:1944 @ 06/26/23 21:38:45.17
------------------------------
••••••••••••••••••

Summarizing 4 Failures:
  [FAIL] [sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [It] should enforce policy based on PodSelector with MatchExpressions[Feature:NetworkPolicy]
  test/e2e/network/netpol/network_legacy.go:1944
  [FAIL] [sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [BeforeEach] should support a 'default-deny-ingress' policy [Feature:NetworkPolicy]
  test/e2e/network/netpol/network_legacy.go:1944
  [FAIL] [sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [It] should enforce policy to allow traffic from pods within server namespace based on PodSelector [Feature:NetworkPolicy]
  test/e2e/network/netpol/network_legacy.go:1944
  [FAIL] [sig-network] NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client [BeforeEach] should enforce multiple ingress policies with ingress allow-all policy taking precedence [Feature:NetworkPolicy]
  test/e2e/network/netpol/network_legacy.go:1944

Ran 91 of 7207 Specs in 1964.800 seconds
FAIL! -- 87 Passed | 4 Failed | 0 Pending | 7116 Skipped

The text was updated successfully, but these errors were encountered:

brb · 2023-06-27T06:00:47Z

Discussion on Slack https://cilium.slack.com/archives/C7PE7V806/p1687790469836489

aojea · 2023-07-03T11:03:54Z

connect: operation not permitted

this is something strange since the operation is happening in the socket namespace?

@brb are network policies or something "influencing" somehow the network namespace of the containers?

brb · 2023-07-03T16:52:40Z

are network policies or something "influencing" somehow the network namespace of the containers?

It should not. Why?

aojea · 2023-07-03T20:19:15Z

OTHER: dial tcp 10.96.245.118:81: connect: operation not permitted

the failures are calls inside the network namespace, that seems like an EPERM trying to "connect" , that is socket level, but I'm speculating here,

other observation is that some pods fail the probes, that execed inside the network namespace to localhost, but I couldn't get any job failing with the debug logs enable to be able to check the agent logs, Joe retried some ones but they passed on the retry, it will be nice to get the logs of one of this failures with the agent in debug mode

lmb · 2023-07-04T08:20:23Z

Got another one: https://github.com/cilium/cilium/actions/runs/5446592740/jobs/9907557850

aojea · 2023-07-04T08:46:39Z

yeah, and same symptoms

2023-07-03T16:28:21.6157582Z   OTHER: dial tcp 10.96.131.241:80: connect: operation not permitted
2023-07-03T16:28:21.6158087Z   OTHER: dial tcp 10.96.131.241:80: connect: operation not permitted
2023-07-03T16:28:21.6158619Z   OTHER: dial tcp 10.96.131.241:80: connect: operation not permitted
2023-07-03T16:28:21.6159022Z   OTHER: dial tcp 10.96.131.241:80: connect: operation not permitted
2023-07-03T16:28:21.6159542Z   OTHER: dial tcp 10.96.131.241:80: connect: operation not permitted
2023-07-03T16:28:21.6159737Z

pods can not dial and ...

2023-07-03T16:28:21.6171116Z Jul 3 16:27:54.632: INFO: stdout: "Name: server-cf97l\nNamespace: network-policy-9834\nPriority: 0\nService Account: default\nNode: cilium-testing-worker/172.18.0.2\nStart Time: Mon, 03 Jul 2023 16:19:16 +0000\nLabels: pod-name=server\nAnnotations: \nStatus: Running\nIP: 10.244.2.65\nIPs:\n IP: 10.244.2.65\nContainers:\n server-container-80:\n Container ID: containerd://6e0406e8347a92e1c45a44fb75d965a7e62aacd227be86bfe454c5e36ec1e952\n Image: registry.k8s.io/e2e-test-images/agnhost:2.43\n Image ID: registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e\n Port: 80/TCP\n Host Port: 0/TCP\n Args:\n porter\n State: Running\n Started: Mon, 03 Jul 2023 16:19:36 +0000\n Ready: False\n Restart Count: 0\n Readiness: exec [/agnhost connect --protocol=tcp --timeout=1s 127.0.0.1:80] delay=0s timeout=1s period=10s #success=1 #failure=3\n Environment:\n SERVE_PORT_80: foo\n Mounts:\n /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zrb9r (ro)\n server-container-81:\n Container ID: containerd://94e7029ed84397d3dd8dfe5dd7966a29019229bad6d60674b0959cad14e85a3f\n Image: registry.k8s.io/e2e-test-images/agnhost:2.43\n Image ID: registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e\n Port: 81/TCP\n Host Port: 0/TCP\n Args:\n porter\n State: Running\n Started: Mon, 03 Jul 2023 16:19:37 +0000\n Ready: True\n Restart Count: 0\n Readiness: exec [/agnhost connect --protocol=tcp --timeout=1s 127.0.0.1:81] delay=0s timeout=1s period=10s #success=1 #failure=3\n Environment:\n SERVE_PORT_81: foo\n Mounts:\n /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zrb9r (ro)\nConditions:\n Type Status\n Initialized True \n Ready False \n ContainersReady False \n PodScheduled True \nVolumes:\n kube-api-access-zrb9r:\n Type: Projected (a volume that contains injected data from multiple sources)\n TokenExpirationSeconds: 3607\n ConfigMapName: kube-root-ca.crt\n ConfigMapOptional: \n DownwardAPI: true\nQoS Class: BestEffort\nNode-Selectors: \nTolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s\n node.kubernetes.io/unreachable:NoExecute op=Exists for 300s\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 8m38s default-scheduler Successfully assigned network-policy-9834/server-cf97l to cilium-testing-worker\n Normal Pulling 8m34s kubelet Pulling image "registry.k8s.io/e2e-test-images/agnhost:2.43"\n Normal Pulled 8m19s kubelet Successfully pulled image "registry.k8s.io/e2e-test-images/agnhost:2.43" in 181.743592ms (14.752680678s including waiting)\n Normal Created 8m19s kubelet Created container server-container-80\n Normal Started 8m18s kubelet Started container server-container-80\n Normal Pulled 8m18s kubelet Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine\n Normal Created 8m18s kubelet Created container server-container-81\n Normal Started 8m17s kubelet Started container server-container-81\n Warning Unhealthy 5m21s (x6 over 8m15s) kubelet Readiness probe failed: command "/agnhost connect --protocol=tcp --timeout=1s 127.0.0.1:80" timed out\n Warning Unhealthy 3m33s (x19 over 8m15s) kubelet Readiness probe failed: command "/agnhost connect --protocol=tcp --timeout=1s 127.0.0.1:81" timed out\n"

Some pods never get ready because the probes fail, the probe is an exec, that means that is executed inside the pod network namespace

Readiness probe failed: command "/agnhost connect --protocol=tcp --timeout=1s 127.0.0.1:81" timed out\n"

that execs a connection inside the network namespace against localhost and fails, so there is something blocking the connections inside the pod namespace, does cilium inject rules inside the cgroup or socket hooks?

aojea · 2023-07-04T13:47:49Z

interestingly the other job for network policies have socketLB disable

cilium/.github/workflows/conformance-k8s-network-policies.yaml

Line 89 in 55a09e5

--set socketLB.enabled=false \

The KIND job to run network policies e2e test is flaky and fail with errors related to problem in the Pod network namespace. To discard that this can be related to an interaction with the Services implementation, run this job disabling this feature. Ref: cilium#26492 Signed-off-by: Antonio Ojea <aojea@google.com>

github-actions · 2023-09-03T01:45:13Z

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

github-actions · 2023-09-17T01:45:29Z

This issue has not seen any activity since it was marked stale.
Closing.

qmonnet added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Jun 26, 2023

YutaroHayakawa mentioned this issue Jul 3, 2023

bgpv1: Unconditionally select node when empty nodeSelector is given #26590

Merged

aojea mentioned this issue Jul 4, 2023

ci: don't use socketLB with network policies #26628

Closed

aojea mentioned this issue Jul 11, 2023

fix kind job with network policy failures #26639

Merged

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Sep 3, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: kubernetes-e2e-net-conformance (ipv4) - NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client #26492

CI: kubernetes-e2e-net-conformance (ipv4) - NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client #26492

qmonnet commented Jun 26, 2023

brb commented Jun 27, 2023

aojea commented Jul 3, 2023

brb commented Jul 3, 2023

aojea commented Jul 3, 2023 •

edited

lmb commented Jul 4, 2023

aojea commented Jul 4, 2023 •

edited

aojea commented Jul 4, 2023

github-actions bot commented Sep 3, 2023

github-actions bot commented Sep 17, 2023

CI: kubernetes-e2e-net-conformance (ipv4) - NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client #26492

CI: kubernetes-e2e-net-conformance (ipv4) - NetworkPolicyLegacy [LinuxOnly] NetworkPolicy between server and client #26492

Comments

qmonnet commented Jun 26, 2023

CI failure

brb commented Jun 27, 2023

aojea commented Jul 3, 2023

brb commented Jul 3, 2023

aojea commented Jul 3, 2023 • edited

lmb commented Jul 4, 2023

aojea commented Jul 4, 2023 • edited

aojea commented Jul 4, 2023

github-actions bot commented Sep 3, 2023

github-actions bot commented Sep 17, 2023

aojea commented Jul 3, 2023 •

edited

aojea commented Jul 4, 2023 •

edited