CI: K8sAgentPerNodeConfigTest Correctly computes config overrides

[maintainer-s-little-helper]

Test Name

K8sAgentPerNodeConfigTest Correctly computes config overrides

Failure Output

FAIL: cilium pre-flight checks failed

Stacktrace

Click to show.

/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:453
cilium pre-flight checks failed
Expected
    <*errors.errorString | 0xc00222ce40>: {
        s: "Cilium validation failed: 4m0s timeout expired: Last polled error: connectivity health is failing: cilium-agent 'cilium-h6v8l': connectivity to path 'k8s2.health-endpoint.secondary-addresses[*].icmp.status' is unhealthy: 'Connection timed out'",
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/config.go:42

Standard Output

Click to show.

Standard Error

Click to show.

10:45:52 STEP: Running BeforeAll block for EntireTestsuite K8sAgentPerNodeConfigTest
10:45:52 STEP: Ensuring the namespace kube-system exists
10:45:52 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
10:45:52 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
10:45:52 STEP: Installing Cilium
10:45:53 STEP: Waiting for Cilium to become ready
10:46:14 STEP: Validating Cilium Installation
10:46:14 STEP: Performing Cilium status preflight check
10:46:14 STEP: Checking whether host EP regenerated
10:46:14 STEP: Performing Cilium health check
10:46:14 STEP: Performing Cilium controllers preflight check
10:46:22 STEP: Performing Cilium service preflight check
10:46:22 STEP: Performing K8s service preflight check
10:46:22 STEP: Cilium is not ready yet: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-gf7x8': Exitcode: 1 
Err: exit status 1
Stdout:
 	 
Stderr:
 	 Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
	 Error: Cannot get status/probe: Put "http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe": dial unix /var/run/cilium/health.sock: connect: no such file or directory
	 
	 command terminated with exit code 1
	 

10:46:22 STEP: Performing Cilium status preflight check
10:46:22 STEP: Performing Cilium health check
10:46:22 STEP: Performing Cilium controllers preflight check
10:46:22 STEP: Checking whether host EP regenerated
10:46:29 STEP: Performing Cilium service preflight check
10:46:29 STEP: Performing K8s service preflight check
10:46:45 STEP: Cilium is not ready yet: host EP is not ready: cilium-agent "cilium-gf7x8" host EP is not in ready state: "regenerating"
10:46:45 STEP: Performing Cilium status preflight check
10:46:45 STEP: Performing Cilium health check
10:46:45 STEP: Performing Cilium controllers preflight check
10:46:45 STEP: Checking whether host EP regenerated
10:46:52 STEP: Performing Cilium service preflight check
10:46:52 STEP: Performing K8s service preflight check
10:47:19 STEP: Cilium is not ready yet: connectivity health is failing: cilium-agent 'cilium-h6v8l': connectivity to path 'k8s2.health-endpoint.secondary-addresses[*].icmp.status' is unhealthy: 'Connection timed out'
10:47:19 STEP: Performing Cilium controllers preflight check
10:47:19 STEP: Performing Cilium health check
10:47:19 STEP: Checking whether host EP regenerated
10:47:19 STEP: Performing Cilium status preflight check
10:47:26 STEP: Performing Cilium service preflight check
10:47:26 STEP: Performing K8s service preflight check
10:47:52 STEP: Performing Cilium controllers preflight check
10:47:52 STEP: Performing Cilium health check
10:47:52 STEP: Checking whether host EP regenerated
10:47:52 STEP: Performing Cilium status preflight check
10:47:59 STEP: Performing Cilium service preflight check
10:47:59 STEP: Performing K8s service preflight check
10:48:26 STEP: Performing Cilium controllers preflight check
10:48:26 STEP: Performing Cilium status preflight check
10:48:26 STEP: Performing Cilium health check
10:48:26 STEP: Checking whether host EP regenerated
10:48:33 STEP: Performing Cilium service preflight check
10:48:33 STEP: Performing K8s service preflight check
10:49:00 STEP: Performing Cilium controllers preflight check
10:49:00 STEP: Checking whether host EP regenerated
10:49:00 STEP: Performing Cilium status preflight check
10:49:00 STEP: Performing Cilium health check
10:49:07 STEP: Performing Cilium service preflight check
10:49:07 STEP: Performing K8s service preflight check
10:49:34 STEP: Performing Cilium controllers preflight check
10:49:34 STEP: Performing Cilium health check
10:49:34 STEP: Performing Cilium status preflight check
10:49:34 STEP: Checking whether host EP regenerated
10:49:41 STEP: Performing Cilium service preflight check
10:49:41 STEP: Performing K8s service preflight check
10:50:08 STEP: Performing Cilium controllers preflight check
10:50:08 STEP: Performing Cilium status preflight check
10:50:08 STEP: Performing Cilium health check
10:50:08 STEP: Checking whether host EP regenerated
FAIL: cilium pre-flight checks failed
Expected
    <*errors.errorString | 0xc00222ce40>: {
        s: "Cilium validation failed: 4m0s timeout expired: Last polled error: connectivity health is failing: cilium-agent 'cilium-h6v8l': connectivity to path 'k8s2.health-endpoint.secondary-addresses[*].icmp.status' is unhealthy: 'Connection timed out'",
    }
to be nil
10:50:14 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|bceee38f_K8sAgentPerNodeConfigTest_Correctly_computes_config_overrides.zip]]
10:50:14 STEP: Running AfterAll block for EntireTestsuite K8sAgentPerNodeConfigTest
10:50:14 STEP: Removing Cilium installation using generated helm manifest
10:50:15 STEP: Cilium is not ready yet: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-gf7x8': Exitcode: 1 
Err: exit status 1
Stdout:
 	 
Stderr:
 	 Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
	 Error: Cannot get status/probe: Put "http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe": EOF
	 
	 command terminated with exit code 1
	 

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1301/artifact/bceee38f_K8sAgentPerNodeConfigTest_Correctly_computes_config_overrides.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//1301/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_1301_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/1301/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

[joestringer]

I also hit this in K8sDatapathLRPTests Checks local redirect policy LRP connectivity in an unrelated PR.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-4.19/53/testReport/junit/Suite-k8s-1/26/K8sDatapathLRPTests_Checks_local_redirect_policy_LRP_connectivity/

When we see various tests failing with a very similar symptom, it can often be caused by some previous test failing to correctly clean up after itself. In the case above it appears to have run after K8sKafkaPolicyTest Kafka Policy Tests and a scattering of other tests that were skipped.

[joestringer]

Also related, this earlier report with similar symptoms: #23307

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has not seen any activity since it was marked stale.
Closing.