Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master #20723

Closed
maintainer-s-little-helper bot opened this issue Jul 31, 2022 · 20 comments
Closed

CI: K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master #20723

maintainer-s-little-helper bot opened this issue Jul 31, 2022 · 20 comments
Assignees
@pchaigno
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me!

Comments

@maintainer-s-little-helper
Copy link

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:465
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0020cea40>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        err: <*errors.errorString | 0xc002063c80>{
            s: "Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:80

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-6c7d4c9fd8-rtxwh      false     false
prometheus-55777f54d9-6lp9x   false     false
coredns-6874cd75d4-8f4sj      false     false

Standard Error

Click to show. 
02:03:18 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
02:03:18 STEP: Ensuring the namespace kube-system exists
02:03:18 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
02:03:18 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
02:03:18 STEP: Deleting pods [echo-694c58bbf4-lwmmp,echo-694c58bbf4-mb2ht] in namespace default
02:03:18 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-lwmmp,echo-694c58bbf4-mb2ht)
02:03:18 STEP: Deleting cep [echo-694c58bbf4-lwmmp,echo-694c58bbf4-mb2ht] in namespace default
02:03:18 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-lwmmp,echo-694c58bbf4-mb2ht)
02:03:28 STEP: Unable to delete pods echo-694c58bbf4-lwmmp with 'kubectl -n default delete pods echo-694c58bbf4-lwmmp': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-lwmmp" deleted
	 
Stderr:
 	 

02:03:28 STEP: Unable to delete pods echo-694c58bbf4-mb2ht with 'kubectl -n default delete pods echo-694c58bbf4-mb2ht': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-mb2ht" deleted
	 
Stderr:
 	 

02:07:29 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0020cea40>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        err: <*errors.errorString | 0xc002063c80>{
            s: "Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        },
    }
to be nil
02:11:29 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
02:11:29 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-6c7d4c9fd8-rtxwh       1/1     Running       0          43m     10.0.0.154      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-55777f54d9-6lp9x    1/1     Running       0          43m     10.0.0.176      k8s1   <none>           <none>
	 default             echo-694c58bbf4-lwmmp          0/2     Terminating   0          9m3s    10.0.0.109      k8s1   <none>           <none>
	 default             echo-694c58bbf4-mb2ht          0/2     Terminating   0          9m3s    10.0.1.95       k8s2   <none>           <none>
	 kube-system         coredns-6874cd75d4-8f4sj       1/1     Running       0          9m15s   10.0.1.147      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-2wrm2               1/1     Running       0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-wpxsd               1/1     Running       0          43m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-5mhhv             1/1     Running       0          43m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-wvvd6             1/1     Running       0          43m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-8mw2k           1/1     Running       0          43m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-bwnpj           1/1     Running       0          43m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
02:11:31 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0006a0280>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        err: <*errors.errorString | 0xc0020840c0>{
            s: "Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        },
    }
to be nil
02:15:31 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|8edded32_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
02:15:31 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
02:15:31 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001b44080>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        err: <*errors.errorString | 0xc001106120>{
            s: "Pods are still terminating: [echo-694c58bbf4-lwmmp echo-694c58bbf4-mb2ht]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//350/artifact/8edded32_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//350/artifact/test_results_Cilium-PR-K8s-1.23-kernel-4.19_350_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19/350/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
@maintainer-s-little-helper maintainer-s-little-helper bot added the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Jul 31, 2022
Merged
@joestringer
Copy link
Member

This appears to be a testsuite issue where Cilium is removed from the cluster before removing the pods, so then when kubelet attempts to delete the pods there is no CNI available to handle the delete request, leading to timeouts and the pods being stuck in Terminating state.

@brb
Copy link
Member

brb commented Aug 2, 2022 •
edited

Weird, I thought that this should have fixed this - #19666 #19750.

@maintainer-s-little-helper
Copy link
Author

PR #20745 hit this flake with 93.42% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:465
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000b02660>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        err: <*errors.errorString | 0xc0012923b0>{
            s: "Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:80

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
coredns-6874cd75d4-zbcmj      false     false
grafana-6c7d4c9fd8-tgtcn      false     false
prometheus-55777f54d9-vng27   false     false

Standard Error

Click to show. 
11:02:54 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
11:02:54 STEP: Ensuring the namespace kube-system exists
11:02:54 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
11:02:54 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
11:02:54 STEP: Deleting cep [echo-694c58bbf4-5cvjv,echo-694c58bbf4-wph29] in namespace default
11:02:54 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-5cvjv,echo-694c58bbf4-wph29)
11:02:54 STEP: Deleting pods [echo-694c58bbf4-5cvjv,echo-694c58bbf4-wph29] in namespace default
11:02:54 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-5cvjv,echo-694c58bbf4-wph29)
11:03:04 STEP: Unable to delete pods echo-694c58bbf4-wph29 with 'kubectl -n default delete pods echo-694c58bbf4-wph29': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-wph29" deleted
	 
Stderr:
 	 

11:03:04 STEP: Unable to delete pods echo-694c58bbf4-5cvjv with 'kubectl -n default delete pods echo-694c58bbf4-5cvjv': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-5cvjv" deleted
	 
Stderr:
 	 

11:07:04 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000b02660>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        err: <*errors.errorString | 0xc0012923b0>{
            s: "Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        },
    }
to be nil
11:11:04 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
11:11:04 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-6c7d4c9fd8-tgtcn       1/1     Running       0          24m     10.0.1.123      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-55777f54d9-vng27    1/1     Running       0          24m     10.0.1.12       k8s1   <none>           <none>
	 default             echo-694c58bbf4-5cvjv          0/2     Terminating   0          9m7s    10.0.1.32       k8s1   <none>           <none>
	 default             echo-694c58bbf4-wph29          0/2     Terminating   0          9m7s    10.0.0.158      k8s2   <none>           <none>
	 kube-system         coredns-6874cd75d4-zbcmj       1/1     Running       0          9m25s   10.0.1.76       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-trs2p               1/1     Running       0          25m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-v6gvl               1/1     Running       0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-5hq8t             1/1     Running       0          24m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-sm2hq             1/1     Running       0          24m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-bbzxz           1/1     Running       0          25m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-w76rm           1/1     Running       0          25m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
11:11:06 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000b02460>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        err: <*errors.errorString | 0xc000ab69f0>{
            s: "Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        },
    }
to be nil
11:15:06 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|25c601f6_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
11:15:06 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
11:15:06 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000fbe2c0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        err: <*errors.errorString | 0xc000ab67b0>{
            s: "Pods are still terminating: [echo-694c58bbf4-5cvjv echo-694c58bbf4-wph29]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//360/artifact/25c601f6_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//360/artifact/test_results_Cilium-PR-K8s-1.23-kernel-4.19_360_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19/360/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Aug 2, 2022
Merged
@maintainer-s-little-helper
Copy link
Author

PR #20859 hit this flake with 94.17% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:465
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0016b2a60>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        err: <*errors.errorString | 0xc000b586b0>{
            s: "Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:80

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
coredns-6874cd75d4-6n97g      false     false
grafana-6c7d4c9fd8-dq4gg      false     false
prometheus-55777f54d9-8c6ds   false     false

Standard Error

Click to show. 
08:18:43 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
08:18:43 STEP: Ensuring the namespace kube-system exists
08:18:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
08:18:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
08:18:43 STEP: Deleting cep [echo-694c58bbf4-d687n,echo-694c58bbf4-w8zfc] in namespace default
08:18:43 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-d687n,echo-694c58bbf4-w8zfc)
08:18:43 STEP: Deleting pods [echo-694c58bbf4-d687n,echo-694c58bbf4-w8zfc] in namespace default
08:18:43 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-d687n,echo-694c58bbf4-w8zfc)
08:18:53 STEP: Unable to delete pods echo-694c58bbf4-d687n with 'kubectl -n default delete pods echo-694c58bbf4-d687n': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-d687n" deleted
	 
Stderr:
 	 

08:18:53 STEP: Unable to delete pods echo-694c58bbf4-w8zfc with 'kubectl -n default delete pods echo-694c58bbf4-w8zfc': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-w8zfc" deleted
	 
Stderr:
 	 

08:22:53 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0016b2a60>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        err: <*errors.errorString | 0xc000b586b0>{
            s: "Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        },
    }
to be nil
08:26:53 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
08:26:53 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-6c7d4c9fd8-dq4gg       1/1     Running       0          25m     10.0.1.241      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-55777f54d9-8c6ds    1/1     Running       0          25m     10.0.1.53       k8s2   <none>           <none>
	 default             echo-694c58bbf4-d687n          0/2     Terminating   0          9m5s    <none>          k8s1   <none>           <none>
	 default             echo-694c58bbf4-w8zfc          0/2     Terminating   0          9m5s    10.0.1.125      k8s2   <none>           <none>
	 kube-system         coredns-6874cd75d4-6n97g       1/1     Running       0          9m29s   10.0.0.54       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-jxmjf               1/1     Running       0          26m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-nqcmw               1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-sgxbh             1/1     Running       0          25m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-snkmb             1/1     Running       0          25m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-8fzq2           1/1     Running       0          26m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-cxpqg           1/1     Running       0          26m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
08:26:56 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0007ae020>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        err: <*errors.errorString | 0xc000b580c0>{
            s: "Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        },
    }
to be nil
08:30:56 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|d1e2f460_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
08:30:56 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
08:30:56 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc00012e2a0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        err: <*errors.errorString | 0xc00080a3f0>{
            s: "Pods are still terminating: [echo-694c58bbf4-d687n echo-694c58bbf4-w8zfc]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//432/artifact/d1e2f460_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//432/artifact/test_results_Cilium-PR-K8s-1.23-kernel-4.19_432_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19/432/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #20969 hit this flake with 93.87% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:465
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000f58580>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        err: <*errors.errorString | 0xc00051e730>{
            s: "Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:80

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-6c7d4c9fd8-vh5rc      false     false
prometheus-55777f54d9-rdpcw   false     false
coredns-6874cd75d4-d5ch8      false     false

Standard Error

Click to show. 
22:19:23 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
22:19:23 STEP: Ensuring the namespace kube-system exists
22:19:23 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
22:19:23 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
22:19:23 STEP: Deleting pods [echo-694c58bbf4-6fd9z,echo-694c58bbf4-d4dkh] in namespace default
22:19:23 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-6fd9z,echo-694c58bbf4-d4dkh)
22:19:23 STEP: Deleting cep [echo-694c58bbf4-6fd9z,echo-694c58bbf4-d4dkh] in namespace default
22:19:23 STEP: Waiting for 2 deletes to return (echo-694c58bbf4-6fd9z,echo-694c58bbf4-d4dkh)
22:19:33 STEP: Unable to delete pods echo-694c58bbf4-d4dkh with 'kubectl -n default delete pods echo-694c58bbf4-d4dkh': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-d4dkh" deleted
	 
Stderr:
 	 

22:19:33 STEP: Unable to delete pods echo-694c58bbf4-6fd9z with 'kubectl -n default delete pods echo-694c58bbf4-6fd9z': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-694c58bbf4-6fd9z" deleted
	 
Stderr:
 	 

22:23:33 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000f58580>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        err: <*errors.errorString | 0xc00051e730>{
            s: "Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        },
    }
to be nil
22:27:33 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
22:27:33 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-6c7d4c9fd8-vh5rc       1/1     Running       0          9m59s   10.0.0.247      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-55777f54d9-rdpcw    1/1     Running       0          9m59s   10.0.0.222      k8s1   <none>           <none>
	 default             echo-694c58bbf4-6fd9z          0/2     Terminating   0          8m45s   10.0.1.175      k8s2   <none>           <none>
	 default             echo-694c58bbf4-d4dkh          0/2     Terminating   0          8m45s   10.0.0.214      k8s1   <none>           <none>
	 kube-system         coredns-6874cd75d4-d5ch8       1/1     Running       0          8m56s   10.0.0.154      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0          13m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0          13m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       0          13m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-7d7bc               1/1     Running       0          12m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-tm2jx               1/1     Running       0          10m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       0          13m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-kfzxl             1/1     Running       0          10m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-mh2l9             1/1     Running       0          10m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-hlddr           1/1     Running       0          10m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-ls5pj           1/1     Running       0          10m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
22:27:35 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0005c5440>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        err: <*errors.errorString | 0xc000812d40>{
            s: "Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        },
    }
to be nil
22:31:35 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|8782d413_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
22:31:35 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
22:31:35 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0005259e0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        err: <*errors.errorString | 0xc000748ff0>{
            s: "Pods are still terminating: [echo-694c58bbf4-6fd9z echo-694c58bbf4-d4dkh]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//503/artifact/8782d413_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//503/artifact/test_results_Cilium-PR-K8s-1.23-kernel-4.19_503_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19/503/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@github-actions
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Oct 19, 2022
@github-actions
Copy link

github-actions bot commented Nov 3, 2022

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as completed Nov 3, 2022
@joestringer joestringer reopened this Jan 24, 2023
This was referenced Jan 24, 2023
Merged
Closed
@github-actions github-actions bot removed the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jan 24, 2023
@maintainer-s-little-helper
Copy link
Author

PR #23284 hit this flake with 87.02% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001090ac0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        err: <*errors.errorString | 0xc0005b72f0>{
            s: "Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-5747bcc8f9-6kczz      false     false
prometheus-655fb888d7-wbb2x   false     false
coredns-69b675786c-hg5rf      false     false

Standard Error

Click to show. 
12:05:06 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
12:05:06 STEP: Ensuring the namespace kube-system exists
12:05:06 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
12:05:06 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
12:05:06 STEP: Deleting pods [echo-55fdf5787d-d4r4q,echo-55fdf5787d-wdrdg] in namespace default
12:05:06 STEP: Waiting for 2 deletes to return (echo-55fdf5787d-d4r4q,echo-55fdf5787d-wdrdg)
12:05:06 STEP: Deleting cep [echo-55fdf5787d-d4r4q,echo-55fdf5787d-wdrdg,test-k8s2-7f96d84c65-csd9b] in namespace default
12:05:06 STEP: Waiting for 3 deletes to return (echo-55fdf5787d-d4r4q,echo-55fdf5787d-wdrdg,test-k8s2-7f96d84c65-csd9b)
12:05:16 STEP: Unable to delete pods echo-55fdf5787d-wdrdg with 'kubectl -n default delete pods echo-55fdf5787d-wdrdg': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-wdrdg" deleted
	 
Stderr:
 	 

12:05:16 STEP: Unable to delete pods echo-55fdf5787d-d4r4q with 'kubectl -n default delete pods echo-55fdf5787d-d4r4q': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-d4r4q" deleted
	 
Stderr:
 	 

12:09:16 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001090ac0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        err: <*errors.errorString | 0xc0005b72f0>{
            s: "Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        },
    }
to be nil
12:13:16 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
12:13:16 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS      AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-6kczz       1/1     Running       0             37m     10.0.1.160      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-wbb2x    1/1     Running       0             37m     10.0.1.130      k8s1   <none>           <none>
	 default             echo-55fdf5787d-d4r4q          0/2     Terminating   0             11m     10.0.0.81       k8s2   <none>           <none>
	 default             echo-55fdf5787d-wdrdg          0/2     Terminating   0             11m     <none>          k8s1   <none>           <none>
	 kube-system         coredns-69b675786c-hg5rf       1/1     Running       0             9m42s   10.0.0.54       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0             42m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0             42m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       3 (37m ago)   42m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-4cxzv               1/1     Running       0             41m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-8724x               1/1     Running       0             38m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       2 (40m ago)   42m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-kn7kp             1/1     Running       0             38m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-nhjws             1/1     Running       0             38m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-2nnvc           1/1     Running       0             38m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-dpvkk           1/1     Running       0             38m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
12:13:19 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000942000>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        err: <*errors.errorString | 0xc0015000c0>{
            s: "Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        },
    }
to be nil
12:17:19 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|e13a9a44_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
12:17:19 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
12:17:19 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0016a8020>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        err: <*errors.errorString | 0xc0017820c0>{
            s: "Pods are still terminating: [echo-55fdf5787d-d4r4q echo-55fdf5787d-wdrdg]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//394/artifact/e13a9a44_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//394/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.9_394_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9/394/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
9 tasks
@kaworu kaworu mentioned this issue Feb 3, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #23520 hit this flake with 87.17% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000996be0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        err: <*errors.errorString | 0xc000994fe0>{
            s: "Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-5747bcc8f9-q59ks      false     false
prometheus-655fb888d7-wrq7n   false     false
coredns-69b675786c-mlkwx      false     false

Standard Error

Click to show. 
09:46:23 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
09:46:23 STEP: Ensuring the namespace kube-system exists
09:46:23 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
09:46:23 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
09:46:23 STEP: Deleting pods [echo-55fdf5787d-7n9ds,echo-55fdf5787d-c4qwv] in namespace default
09:46:23 STEP: Waiting for 2 deletes to return (echo-55fdf5787d-7n9ds,echo-55fdf5787d-c4qwv)
09:46:33 STEP: Unable to delete pods echo-55fdf5787d-c4qwv with 'kubectl -n default delete pods echo-55fdf5787d-c4qwv': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-c4qwv" deleted
	 
Stderr:
 	 

09:46:33 STEP: Unable to delete pods echo-55fdf5787d-7n9ds with 'kubectl -n default delete pods echo-55fdf5787d-7n9ds': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-7n9ds" deleted
	 
Stderr:
 	 

09:50:33 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000996be0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        err: <*errors.errorString | 0xc000994fe0>{
            s: "Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        },
    }
to be nil
09:54:33 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
09:54:33 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS      AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-q59ks       1/1     Running       0             15m     10.0.0.22       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-wrq7n    1/1     Running       0             15m     10.0.0.186      k8s2   <none>           <none>
	 default             echo-55fdf5787d-7n9ds          0/2     Terminating   0             11m     10.0.0.166      k8s2   <none>           <none>
	 default             echo-55fdf5787d-c4qwv          0/2     Terminating   0             11m     10.0.1.55       k8s1   <none>           <none>
	 kube-system         coredns-69b675786c-mlkwx       1/1     Running       0             9m53s   10.0.1.82       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0             19m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0             19m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       3 (14m ago)   19m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-9f7c6               1/1     Running       0             19m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-c77gt               1/1     Running       0             16m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       3 (14m ago)   19m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-n8pxf             1/1     Running       0             15m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-qcmnn             1/1     Running       0             15m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-4wtm7           1/1     Running       0             16m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-zfp9l           1/1     Running       0             16m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
09:54:35 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0008360a0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        err: <*errors.errorString | 0xc0000ad090>{
            s: "Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        },
    }
to be nil
09:58:35 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|f4e36e32_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
09:58:36 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
09:58:36 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc00053a480>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        err: <*errors.errorString | 0xc00040c0e0>{
            s: "Pods are still terminating: [echo-55fdf5787d-7n9ds echo-55fdf5787d-c4qwv]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//436/artifact/f4e36e32_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//436/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.9_436_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9/436/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Feb 7, 2023
Merged
1 task
@pchaigno pchaigno mentioned this issue Feb 12, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #23687 hit this flake with 85.95% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000fc65e0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        err: <*errors.errorString | 0xc00264e8b0>{
            s: "Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-5747bcc8f9-9qzpm      false     false
prometheus-655fb888d7-6cjsh   false     false
coredns-69b675786c-x62ww      false     false

Standard Error

Click to show. 
14:37:35 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
14:37:35 STEP: Ensuring the namespace kube-system exists
14:37:35 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
14:37:35 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
14:37:35 STEP: Deleting cep [echo-55fdf5787d-5sp97,echo-55fdf5787d-cdvvs,test-k8s2-7f96d84c65-xjhcl] in namespace default
14:37:35 STEP: Waiting for 3 deletes to return (echo-55fdf5787d-5sp97,echo-55fdf5787d-cdvvs,test-k8s2-7f96d84c65-xjhcl)
14:37:35 STEP: Deleting pods [echo-55fdf5787d-5sp97,echo-55fdf5787d-cdvvs,test-k8s2-7f96d84c65-xjhcl] in namespace default
14:37:35 STEP: Waiting for 3 deletes to return (echo-55fdf5787d-5sp97,echo-55fdf5787d-cdvvs,test-k8s2-7f96d84c65-xjhcl)
14:37:45 STEP: Unable to delete pods echo-55fdf5787d-cdvvs with 'kubectl -n default delete pods echo-55fdf5787d-cdvvs': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-cdvvs" deleted
	 
Stderr:
 	 

14:37:45 STEP: Unable to delete pods echo-55fdf5787d-5sp97 with 'kubectl -n default delete pods echo-55fdf5787d-5sp97': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-5sp97" deleted
	 
Stderr:
 	 

14:41:45 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000fc65e0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        err: <*errors.errorString | 0xc00264e8b0>{
            s: "Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        },
    }
to be nil
14:45:45 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
14:45:45 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS      AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-9qzpm       1/1     Running       0             22m     10.0.1.79       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-6cjsh    1/1     Running       0             22m     10.0.1.252      k8s2   <none>           <none>
	 default             echo-55fdf5787d-5sp97          0/2     Terminating   0             11m     10.0.1.156      k8s2   <none>           <none>
	 default             echo-55fdf5787d-cdvvs          0/2     Terminating   0             11m     10.0.0.75       k8s1   <none>           <none>
	 kube-system         coredns-69b675786c-x62ww       1/1     Running       0             9m41s   10.0.1.69       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0             26m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0             26m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       2 (21m ago)   26m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-nnkzg               1/1     Running       0             26m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-v8nwt               1/1     Running       0             23m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       3 (21m ago)   26m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-m92gg             1/1     Running       0             22m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-n9pb8             1/1     Running       0             22m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-h99cc           1/1     Running       0             23m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-mnzf8           1/1     Running       0             23m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
14:45:48 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0007a2000>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        err: <*errors.errorString | 0xc0003d58a0>{
            s: "Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        },
    }
to be nil
14:49:48 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|44216efc_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
14:49:48 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
14:49:48 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0005a6000>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        err: <*errors.errorString | 0xc00264e0b0>{
            s: "Pods are still terminating: [echo-55fdf5787d-5sp97 echo-55fdf5787d-cdvvs]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//448/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.9_448_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9/448/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
29 tasks
@maintainer-s-little-helper
Copy link
Author

PR #23319 hit this flake with 85.80% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0009811e0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        err: <*errors.errorString | 0xc000f27cb0>{
            s: "Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
prometheus-655fb888d7-z8ccj   false     false
coredns-69b675786c-tp5j2      false     false
grafana-5747bcc8f9-vxn48      false     false

Standard Error

Click to show. 
16:55:29 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
16:55:29 STEP: Ensuring the namespace kube-system exists
16:55:29 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
16:55:29 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
16:55:29 STEP: Deleting pods [echo-55fdf5787d-6mv42,echo-55fdf5787d-cvskl] in namespace default
16:55:29 STEP: Waiting for 2 deletes to return (echo-55fdf5787d-6mv42,echo-55fdf5787d-cvskl)
16:55:39 STEP: Unable to delete pods echo-55fdf5787d-cvskl with 'kubectl -n default delete pods echo-55fdf5787d-cvskl': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-cvskl" deleted
	 
Stderr:
 	 

16:55:39 STEP: Unable to delete pods echo-55fdf5787d-6mv42 with 'kubectl -n default delete pods echo-55fdf5787d-6mv42': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-6mv42" deleted
	 
Stderr:
 	 

16:59:39 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0009811e0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        err: <*errors.errorString | 0xc000f27cb0>{
            s: "Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        },
    }
to be nil
17:03:39 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
17:03:39 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS      AGE   IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-vxn48       1/1     Running       0             31m   10.0.0.247      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-z8ccj    1/1     Running       0             31m   10.0.0.44       k8s2   <none>           <none>
	 default             echo-55fdf5787d-6mv42          0/2     Terminating   0             11m   10.0.1.31       k8s1   <none>           <none>
	 default             echo-55fdf5787d-cvskl          0/2     Terminating   0             11m   10.0.0.230      k8s2   <none>           <none>
	 kube-system         coredns-69b675786c-tp5j2       1/1     Running       0             10m   10.0.1.240      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0             36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0             36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       4 (29m ago)   36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-8k6ws               1/1     Running       0             36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-tqz4r               1/1     Running       0             32m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       4 (29m ago)   36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-hdtf9             1/1     Running       0             31m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-ljww8             1/1     Running       0             31m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-mrz6t           1/1     Running       0             32m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-wtzbf           1/1     Running       0             32m   192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
17:03:42 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0007c2760>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        err: <*errors.errorString | 0xc001014ab0>{
            s: "Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        },
    }
to be nil
17:07:42 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|5f56c9c7_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
17:07:42 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
17:07:42 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc00089da60>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        err: <*errors.errorString | 0xc0005b0110>{
            s: "Pods are still terminating: [echo-55fdf5787d-6mv42 echo-55fdf5787d-cvskl]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//458/artifact/5f56c9c7_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//458/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.9_458_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9/458/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #23319 hit this flake with 85.65% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0002a7cc0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        err: <*errors.errorString | 0xc002c37760>{
            s: "Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-5747bcc8f9-bpmbj      false     false
prometheus-655fb888d7-k8ltz   false     false
coredns-69b675786c-jj8q2      false     false

Standard Error

Click to show. 
15:28:45 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
15:28:45 STEP: Ensuring the namespace kube-system exists
15:28:45 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
15:28:45 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
15:28:45 STEP: Deleting cep [echo-55fdf5787d-dmg9n,echo-55fdf5787d-m2lf6,test-k8s2-7f96d84c65-962bt] in namespace default
15:28:45 STEP: Waiting for 3 deletes to return (echo-55fdf5787d-dmg9n,echo-55fdf5787d-m2lf6,test-k8s2-7f96d84c65-962bt)
15:28:45 STEP: Deleting pods [echo-55fdf5787d-dmg9n,echo-55fdf5787d-m2lf6] in namespace default
15:28:45 STEP: Waiting for 2 deletes to return (echo-55fdf5787d-dmg9n,echo-55fdf5787d-m2lf6)
15:28:55 STEP: Unable to delete pods echo-55fdf5787d-m2lf6 with 'kubectl -n default delete pods echo-55fdf5787d-m2lf6': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-m2lf6" deleted
	 
Stderr:
 	 

15:28:55 STEP: Unable to delete pods echo-55fdf5787d-dmg9n with 'kubectl -n default delete pods echo-55fdf5787d-dmg9n': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-dmg9n" deleted
	 
Stderr:
 	 

15:32:55 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0002a7cc0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        err: <*errors.errorString | 0xc002c37760>{
            s: "Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        },
    }
to be nil
15:36:55 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
15:36:55 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS      AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-bpmbj       1/1     Running       0             32m     10.0.1.73       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-k8ltz    1/1     Running       0             32m     10.0.1.113      k8s2   <none>           <none>
	 default             echo-55fdf5787d-dmg9n          0/2     Terminating   0             11m     10.0.1.119      k8s2   <none>           <none>
	 default             echo-55fdf5787d-m2lf6          0/2     Terminating   0             11m     10.0.0.221      k8s1   <none>           <none>
	 kube-system         coredns-69b675786c-jj8q2       1/1     Running       0             9m45s   10.0.1.163      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0             37m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0             37m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       4 (29m ago)   37m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-7fnhw               1/1     Running       0             33m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-sgw8d               1/1     Running       0             34m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       4 (29m ago)   37m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-rw8rb             1/1     Running       0             32m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-tk9lw             1/1     Running       0             32m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-7chrp           1/1     Running       0             33m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-tj46x           1/1     Running       0             33m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
15:36:57 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc002b88020>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        err: <*errors.errorString | 0xc0008042b0>{
            s: "Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        },
    }
to be nil
15:40:57 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|7d695e14_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
15:40:57 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
15:40:57 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0004b0200>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        err: <*errors.errorString | 0xc0007de230>{
            s: "Pods are still terminating: [echo-55fdf5787d-dmg9n echo-55fdf5787d-m2lf6]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//460/artifact/1292748b_K8sAgentIstioTest_Istio_Bookinfo_Demo_Tests_bookinfo_inter-service_connectivity.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//460/artifact/7d695e14_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//460/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.9_460_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9/460/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #23851 hit this flake with 85.46% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001b793a0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        err: <*errors.errorString | 0xc000539f90>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-4.9/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-84476dcf4b-zwj8f      false     false
prometheus-7dbb447479-rn2ld   false     false
coredns-8c79ffd8b-cbgl2       false     false

Standard Error

Click to show. 
14:45:05 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
14:45:05 STEP: Ensuring the namespace kube-system exists
14:45:05 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
14:45:05 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
14:45:05 STEP: Deleting pods [echo-bc5cf6ffb-pkkb5,echo-bc5cf6ffb-wx7tc] in namespace default
14:45:05 STEP: Waiting for 2 deletes to return (echo-bc5cf6ffb-pkkb5,echo-bc5cf6ffb-wx7tc)
14:45:15 STEP: Unable to delete pods echo-bc5cf6ffb-pkkb5 with 'kubectl -n default delete pods echo-bc5cf6ffb-pkkb5': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-bc5cf6ffb-pkkb5" deleted
	 
Stderr:
 	 

14:45:15 STEP: Unable to delete pods echo-bc5cf6ffb-wx7tc with 'kubectl -n default delete pods echo-bc5cf6ffb-wx7tc': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-bc5cf6ffb-wx7tc" deleted
	 
Stderr:
 	 

14:49:15 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001b793a0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        err: <*errors.errorString | 0xc000539f90>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        },
    }
to be nil
14:53:15 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
14:53:15 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS      AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-84476dcf4b-zwj8f       1/1     Running       0             23m     10.0.1.164      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-7dbb447479-rn2ld    1/1     Running       0             23m     10.0.1.40       k8s2   <none>           <none>
	 default             echo-bc5cf6ffb-pkkb5           0/2     Terminating   0             11m     10.0.0.34       k8s1   <none>           <none>
	 default             echo-bc5cf6ffb-wx7tc           0/2     Terminating   0             11m     10.0.1.227      k8s2   <none>           <none>
	 kube-system         coredns-8c79ffd8b-cbgl2        1/1     Running       0             9m49s   10.0.0.86       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0             28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0             28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       4 (15m ago)   28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-4bl95               1/1     Running       0             27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-dpvwd               1/1     Running       0             24m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       4 (15m ago)   28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-bxtd4             1/1     Running       0             23m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-csljq             1/1     Running       0             23m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-4mk6v           1/1     Running       0             24m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-wdvhr           1/1     Running       0             24m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
14:53:17 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000813500>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        err: <*errors.errorString | 0xc0005f0710>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        },
    }
to be nil
14:57:17 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|699b420e_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
14:57:17 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
14:57:17 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000e81940>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        err: <*errors.errorString | 0xc0015acc90>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-pkkb5 echo-bc5cf6ffb-wx7tc]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.9//56/artifact/699b420e_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.9//56/artifact/test_results_Cilium-PR-K8s-1.24-kernel-4.9_56_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.9/56/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
1 task
@maintainer-s-little-helper
Copy link
Author

PR #24009 hit this flake with 88.21% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000a65720>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [cilium-gk4v6]",
        err: <*errors.errorString | 0xc002258610>{
            s: "Pods are still terminating: [cilium-gk4v6]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: [cilium-gk4v6]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
coredns-6b775575b5-4t9zf      false     false
grafana-84476dcf4b-6q48l      false     false
prometheus-7dbb447479-ksd8n   false     false

Standard Error

Click to show. 
10:45:31 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
10:45:31 STEP: Ensuring the namespace kube-system exists
10:45:31 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
10:45:31 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
10:45:31 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000a65720>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [cilium-gk4v6]",
        err: <*errors.errorString | 0xc002258610>{
            s: "Pods are still terminating: [cilium-gk4v6]",
        },
    }
to be nil
10:49:31 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
10:49:31 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Err: context deadline exceeded
Stdout:
 	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
10:59:32 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0000bdd20>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [cilium-gk4v6]",
        err: <*errors.errorString | 0xc001a066d0>{
            s: "Pods are still terminating: [cilium-gk4v6]",
        },
    }
to be nil
11:03:32 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|4973eb61_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
11:03:33 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
11:03:33 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0001ca200>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [cilium-gk4v6]",
        err: <*errors.errorString | 0xc001a066e0>{
            s: "Pods are still terminating: [cilium-gk4v6]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/07b2ebb7_K8sAgentHubbleTest_Hubble_Observe_Test_L7_Flow_with_hubble-relay.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/39f17e9c_K8sAgentHubbleTest_Hubble_Observe_Test_L3-L4_Flow.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/42000fce_K8sAgentHubbleTest_Hubble_Observe_Test_L3-L4_Flow_with_hubble-relay.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/4973eb61_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/6972d528_K8sPolicyTestExtended_Validate_toEntities_KubeAPIServer_Still_allows_connection_to_KubeAPIServer_with_a_duplicate_policy.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/9f26c82d_K8sAgentHubbleTest_Hubble_Observe_Test_FQDN_Policy_with_Relay.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/a7eee098_K8sAgentHubbleTest_Hubble_Observe_Test_TLS_certificate.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/b49b2687_K8sAgentHubbleTest_Hubble_Observe_Test_L7_Flow.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/f40b8c04_K8sPolicyTestExtended_Validate_toEntities_KubeAPIServer_Denies_connection_to_KubeAPIServer.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1031/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_1031_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/1031/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Mar 1, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #24184 hit this flake with 86.56% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000e191e0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        err: <*errors.errorString | 0xc000c462c0>{
            s: "Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.9/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
prometheus-669755c8c5-6qsc8   false     false
coredns-85fbf8f7dd-8vplx      false     false
grafana-698dc95f6c-bt48m      false     false

Standard Error

Click to show. 
13:57:26 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
13:57:26 STEP: Ensuring the namespace kube-system exists
13:57:26 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
13:57:26 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
13:57:27 STEP: Deleting cep [echo-55fdf5787d-72rsb,echo-55fdf5787d-dxjrk,test-k8s2-7f96d84c65-sgw7x] in namespace default
13:57:27 STEP: Waiting for 3 deletes to return (echo-55fdf5787d-72rsb,echo-55fdf5787d-dxjrk,test-k8s2-7f96d84c65-sgw7x)
13:57:27 STEP: Deleting pods [echo-55fdf5787d-72rsb,echo-55fdf5787d-dxjrk] in namespace default
13:57:27 STEP: Waiting for 2 deletes to return (echo-55fdf5787d-72rsb,echo-55fdf5787d-dxjrk)
13:57:37 STEP: Unable to delete pods echo-55fdf5787d-dxjrk with 'kubectl -n default delete pods echo-55fdf5787d-dxjrk': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-dxjrk" deleted
	 
Stderr:
 	 

13:57:37 STEP: Unable to delete pods echo-55fdf5787d-72rsb with 'kubectl -n default delete pods echo-55fdf5787d-72rsb': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-55fdf5787d-72rsb" deleted
	 
Stderr:
 	 

14:01:37 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000e191e0>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        err: <*errors.errorString | 0xc000c462c0>{
            s: "Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        },
    }
to be nil
14:05:37 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
14:05:37 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS      AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-698dc95f6c-bt48m       1/1     Running       0             15m     10.0.1.133      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-669755c8c5-6qsc8    1/1     Running       0             15m     10.0.1.8        k8s1   <none>           <none>
	 default             echo-55fdf5787d-72rsb          0/2     Terminating   0             11m     10.0.0.96       k8s2   <none>           <none>
	 default             echo-55fdf5787d-dxjrk          0/2     Terminating   0             11m     10.0.1.50       k8s1   <none>           <none>
	 kube-system         coredns-85fbf8f7dd-8vplx       1/1     Running       0             9m44s   10.0.1.192      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0             20m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0             20m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       2 (18m ago)   20m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-bqdgf               1/1     Running       0             16m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-gvclw               1/1     Running       0             19m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       2 (18m ago)   20m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-4dl4b             1/1     Running       0             16m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-5sxvf             1/1     Running       0             16m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-pwrkc           1/1     Running       0             16m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-xqv2k           1/1     Running       0             16m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
14:05:39 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000db7180>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        err: <*errors.errorString | 0xc000c473a0>{
            s: "Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        },
    }
to be nil
14:09:39 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|630c5279_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
14:09:39 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
14:09:39 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001b9d580>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        err: <*errors.errorString | 0xc000413b30>{
            s: "Pods are still terminating: [echo-55fdf5787d-72rsb echo-55fdf5787d-dxjrk]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//488/artifact/630c5279_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9//488/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.9_488_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.9/488/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Mar 6, 2023
Merged
2 tasks
@lmb
Copy link
Contributor

lmb commented Mar 14, 2023

Another one https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next/1266/consoleFull

@maintainer-s-little-helper
Copy link
Author

PR #23606 hit this flake with 86.39% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0004b0f60>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        err: <*errors.errorString | 0xc0012614e0>{
            s: "Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-98b4b9789-d8whx       false     false
prometheus-6f66c554f4-7mgnj   false     false
coredns-567b6dd84-mvfl6       false     false

Standard Error

Click to show. 
11:36:30 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
11:36:30 STEP: Ensuring the namespace kube-system exists
11:36:30 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
11:36:30 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
11:36:31 STEP: Deleting cep [echo-9674cb9d4-j5snq,echo-9674cb9d4-l78r5] in namespace default
11:36:31 STEP: Waiting for 2 deletes to return (echo-9674cb9d4-j5snq,echo-9674cb9d4-l78r5)
11:36:31 STEP: Deleting pods [echo-9674cb9d4-j5snq,echo-9674cb9d4-l78r5] in namespace default
11:36:31 STEP: Waiting for 2 deletes to return (echo-9674cb9d4-j5snq,echo-9674cb9d4-l78r5)
11:36:41 STEP: Unable to delete pods echo-9674cb9d4-j5snq with 'kubectl -n default delete pods echo-9674cb9d4-j5snq': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-9674cb9d4-j5snq" deleted
	 
Stderr:
 	 

11:36:41 STEP: Unable to delete pods echo-9674cb9d4-l78r5 with 'kubectl -n default delete pods echo-9674cb9d4-l78r5': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-9674cb9d4-l78r5" deleted
	 
Stderr:
 	 

11:40:41 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0004b0f60>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        err: <*errors.errorString | 0xc0012614e0>{
            s: "Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        },
    }
to be nil
11:44:41 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
11:44:41 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS   AGE   IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-98b4b9789-d8whx        1/1     Running       0          23m   10.0.0.143      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-6f66c554f4-7mgnj    1/1     Running       0          23m   10.0.0.199      k8s1   <none>           <none>
	 default             echo-9674cb9d4-j5snq           0/2     Terminating   0          13m   10.0.1.162      k8s2   <none>           <none>
	 default             echo-9674cb9d4-l78r5           0/2     Terminating   0          13m   10.0.0.148      k8s1   <none>           <none>
	 kube-system         coredns-567b6dd84-mvfl6        1/1     Running       0          28m   10.0.0.4        k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0          28m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0          28m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       0          28m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-5zpx6               1/1     Running       0          28m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-zpnx6               1/1     Running       0          24m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       0          28m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-2j7vj             1/1     Running       0          23m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-8x9ck             1/1     Running       0          23m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-7qbj8           1/1     Running       0          24m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-f4gxr           1/1     Running       0          24m   192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
11:44:44 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000c5b700>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        err: <*errors.errorString | 0xc0016120d0>{
            s: "Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        },
    }
to be nil
11:48:44 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|860d0119_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
11:48:44 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
11:48:44 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc000c5b620>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        err: <*errors.errorString | 0xc00077e830>{
            s: "Pods are still terminating: [echo-9674cb9d4-j5snq echo-9674cb9d4-l78r5]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//1360/artifact/394559b1_K8sAgentFQDNTest_Validate_that_multiple_specs_are_working_correctly.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//1360/artifact/a26e186a_K8sAgentPolicyTest_Basic_Test_Traffic_redirections_to_proxy_Tests_DNS_proxy_visibility_without_policy.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//1360/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_1360_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/1360/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
7 tasks
@maintainer-s-little-helper
Copy link
Author

PR #24484 hit this flake with 85.78% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0016fb420>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        err: <*errors.errorString | 0xc0003cddb0>{
            s: "Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-98b4b9789-7kl8h       false     false
prometheus-6f66c554f4-mn2tv   false     false
coredns-567b6dd84-zmth4       false     false

Standard Error

Click to show. 
06:58:31 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
06:58:31 STEP: Ensuring the namespace kube-system exists
06:58:31 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
06:58:32 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
06:58:32 STEP: Deleting cep [echo-9674cb9d4-xqwv2,echo-9674cb9d4-z7ksc] in namespace default
06:58:32 STEP: Waiting for 2 deletes to return (echo-9674cb9d4-xqwv2,echo-9674cb9d4-z7ksc)
06:58:32 STEP: Deleting pods [echo-9674cb9d4-xqwv2,echo-9674cb9d4-z7ksc] in namespace default
06:58:32 STEP: Waiting for 2 deletes to return (echo-9674cb9d4-xqwv2,echo-9674cb9d4-z7ksc)
06:58:42 STEP: Unable to delete pods echo-9674cb9d4-xqwv2 with 'kubectl -n default delete pods echo-9674cb9d4-xqwv2': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-9674cb9d4-xqwv2" deleted
	 
Stderr:
 	 

06:58:42 STEP: Unable to delete pods echo-9674cb9d4-z7ksc with 'kubectl -n default delete pods echo-9674cb9d4-z7ksc': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-9674cb9d4-z7ksc" deleted
	 
Stderr:
 	 

07:02:42 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0016fb420>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        err: <*errors.errorString | 0xc0003cddb0>{
            s: "Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        },
    }
to be nil
07:06:42 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
07:06:42 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS   AGE   IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-98b4b9789-7kl8h        1/1     Running       0          60m   10.0.0.250      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-6f66c554f4-mn2tv    1/1     Running       0          60m   10.0.0.253      k8s1   <none>           <none>
	 default             echo-9674cb9d4-xqwv2           0/2     Terminating   0          14m   10.0.0.81       k8s1   <none>           <none>
	 default             echo-9674cb9d4-z7ksc           0/2     Terminating   0          14m   10.0.1.247      k8s2   <none>           <none>
	 kube-system         coredns-567b6dd84-zmth4        1/1     Running       0          23m   10.0.0.42       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0          64m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0          64m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       0          64m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-gjzl8               1/1     Running       0          61m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-q7wdz               1/1     Running       0          64m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       0          64m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-fwnzb             1/1     Running       0          60m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-thr7m             1/1     Running       0          60m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-ddj2r           1/1     Running       0          61m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-kjq7p           1/1     Running       0          61m   192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
07:06:45 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0006a8240>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        err: <*errors.errorString | 0xc000702af0>{
            s: "Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        },
    }
to be nil
07:10:45 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|1b3a920b_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
07:10:45 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
07:10:45 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc0006a8260>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        err: <*errors.errorString | 0xc000478910>{
            s: "Pods are still terminating: [echo-9674cb9d4-xqwv2 echo-9674cb9d4-z7ksc]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//1380/artifact/1b3a920b_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//1380/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_1380_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/1380/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #25077 hit this flake with 87.00% similarity:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001e01c20>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        err: <*errors.errorString | 0xc001b25d40>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        },
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-4.19/src/github.com/cilium/cilium/test/k8s/updates.go:82

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Cilium pods: []
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-84476dcf4b-c9z6w      false     false
prometheus-7dbb447479-zpbzx   false     false
coredns-6b775575b5-gjxtw      false     false

Standard Error

Click to show. 
17:01:25 STEP: Running BeforeAll block for EntireTestsuite K8sUpdates
17:01:25 STEP: Ensuring the namespace kube-system exists
17:01:25 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
17:01:25 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
17:01:25 STEP: Deleting cep [echo-bc5cf6ffb-7jkjd,echo-bc5cf6ffb-d4tjt] in namespace default
17:01:25 STEP: Waiting for 2 deletes to return (echo-bc5cf6ffb-7jkjd,echo-bc5cf6ffb-d4tjt)
17:01:25 STEP: Deleting pods [echo-bc5cf6ffb-7jkjd,echo-bc5cf6ffb-d4tjt] in namespace default
17:01:25 STEP: Waiting for 2 deletes to return (echo-bc5cf6ffb-7jkjd,echo-bc5cf6ffb-d4tjt)
17:01:35 STEP: Unable to delete pods echo-bc5cf6ffb-7jkjd with 'kubectl -n default delete pods echo-bc5cf6ffb-7jkjd': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-bc5cf6ffb-7jkjd" deleted
	 
Stderr:
 	 

17:01:35 STEP: Unable to delete pods echo-bc5cf6ffb-d4tjt with 'kubectl -n default delete pods echo-bc5cf6ffb-d4tjt': Exitcode: -1 
Err: signal: killed
Stdout:
 	 pod "echo-bc5cf6ffb-d4tjt" deleted
	 
Stderr:
 	 

17:05:35 STEP: Waiting for pods to be terminated
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc001e01c20>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        err: <*errors.errorString | 0xc001b25d40>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        },
    }
to be nil
17:09:35 STEP: Running JustAfterEach block for EntireTestsuite K8sUpdates
===================== TEST FAILED =====================
17:09:35 STEP: Running AfterFailed block for EntireTestsuite K8sUpdates
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                           READY   STATUS        RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-84476dcf4b-c9z6w       1/1     Running       0          59m     10.0.0.237      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7dbb447479-zpbzx    1/1     Running       0          59m     10.0.0.2        k8s1   <none>           <none>
	 default             echo-bc5cf6ffb-7jkjd           0/2     Terminating   0          12m     10.0.0.232      k8s1   <none>           <none>
	 default             echo-bc5cf6ffb-d4tjt           0/2     Terminating   0          12m     10.0.1.208      k8s2   <none>           <none>
	 kube-system         coredns-6b775575b5-gjxtw       1/1     Running       0          9m41s   10.0.0.233      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                      1/1     Running       0          63m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1            1/1     Running       0          63m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1   1/1     Running       0          63m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-29f7r               1/1     Running       0          60m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-jh25t               1/1     Running       0          63m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1            1/1     Running       0          63m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-5hw9z             1/1     Running       0          59m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-d2h5h             1/1     Running       0          59m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-5jqzf           1/1     Running       0          60m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-sbjsz           1/1     Running       0          60m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods []
===================== Exiting AfterFailed =====================
17:09:38 STEP: Running AfterEach for block EntireTestsuite K8sUpdates
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc00078f140>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        err: <*errors.errorString | 0xc00161b1b0>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        },
    }
to be nil
17:13:38 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|b4b767a2_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip]]
17:13:38 STEP: Running AfterAll block for EntireTestsuite K8sUpdates
17:13:38 STEP: Cleaning up Cilium components
FAIL: terminating containers are not deleted after timeout
Expected
    <*fmt.wrapError | 0xc002635a60>: {
        msg: "Pods are still not deleted after a timeout: 4m0s timeout expired: Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        err: <*errors.errorString | 0xc0006377f0>{
            s: "Pods are still terminating: [echo-bc5cf6ffb-7jkjd echo-bc5cf6ffb-d4tjt]",
        },
    }
to be nil

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19//12/artifact/b4b767a2_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19//12/artifact/c6b62356_K8sDatapathConfig_Etcd_Check_connectivity.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19//12/artifact/test_results_Cilium-PR-K8s-1.24-kernel-4.19_12_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19/12/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
1 task
@pchaigno
Copy link
Member

This is happening when K8sUpdates runs after a test that removed Cilium. Since one of the first thing that K8sUpdates does is removing a bunch of pods, that hands on the missing CNI. #25002 fixed this indirectly by fixing the order in which we run the tests. I'm unsure it's worth sending a better fix for this.

As a result, we should now only see this in backport PRs. I'll mark #25002 for backport to v1.13 to fix it there as well.

@pchaigno pchaigno mentioned this issue Apr 25, 2023
Merged
@pchaigno pchaigno self-assigned this Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pchaigno pchaigno

Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me!
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@brb @lmb @joestringer @pchaigno