Connectivity test fails with Envoy as DaemonSet

[Smana]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

When deploying Cilium on an EKS cluster AND configuring it with Envoy as DaemonSet the cilium connectivity test command fails.

Here is an extract of the output (let me know if you need the whole output):

  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-to-fqdns-one.one.one.one' to namespace 'cilium-test'..
  [-] Scenario [to-fqdns/pod-to-world]
  [.] Action [to-fqdns/pod-to-world/http-to-one.one.one.one-0: cilium-test/client-6b4b857d98-b2mts (10.0.12.196) -> one.one.one.one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 --retry 3 --retry-all-errors --retry-delay 3 http://one.one.one.one:80" failed: command terminated with exit code 22
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client-6b4b857d98-b2mts during action http-to-one.one.one.one-0
  📄 No flows recorded for peer one.one.one.one-http during action http-to-one.one.one.one-0
  [.] Action [to-fqdns/pod-to-world/https-to-one.one.one.one-0: cilium-test/client-6b4b857d98-b2mts (10.0.12.196) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [to-fqdns/pod-to-world/https-to-one.one.one.one-index-0: cilium-test/client-6b4b857d98-b2mts (10.0.12.196) -> one.one.one.one-https-index (one.one.one.one:443)]
  [.] Action [to-fqdns/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-646b88fb9b-xsb7z (10.0.12.156) -> one.one.one.one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 --retry 3 --retry-all-errors --retry-delay 3 http://one.one.one.one:80" failed: command terminated with exit code 22
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client2-646b88fb9b-xsb7z during action http-to-one.one.one.one-1
  📄 No flows recorded for peer one.one.one.one-http during action http-to-one.one.one.one-1
  [.] Action [to-fqdns/pod-to-world/https-to-one.one.one.one-1: cilium-test/client2-646b88fb9b-xsb7z (10.0.12.156) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [to-fqdns/pod-to-world/https-to-one.one.one.one-index-1: cilium-test/client2-646b88fb9b-xsb7z (10.0.12.156) -> one.one.one.one-https-index (one.one.one.one:443)]
  [-] Scenario [to-fqdns/pod-to-world-2]
  [.] Action [to-fqdns/pod-to-world-2/https-cilium-io-0: cilium-test/client-6b4b857d98-b2mts (10.0.12.196) -> cilium-io-https (cilium.io:443)]
  [.] Action [to-fqdns/pod-to-world-2/https-cilium-io-1: cilium-test/client2-646b88fb9b-xsb7z (10.0.12.156) -> cilium-io-https (cilium.io:443)]
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-to-fqdns-one.one.one.one' from namespace 'cilium-test'..

📋 Test Report
❌ 4/42 tests failed (4/284 actions), 13 tests skipped, 0 scenarios skipped:
Test [client-ingress]:
Test [client-egress-l7]:
  ❌ client-egress-l7/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-646b88fb9b-xsb7z (10.0.12.156) -> one.one.one.one-http (one.one.one.one:80)
Test [client-egress-l7-named-port]:
  ❌ client-egress-l7-named-port/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-646b88fb9b-xsb7z (10.0.12.156) -> one.one.one.one-http (one.one.one.one:80)
Test [to-fqdns]:
  ❌ to-fqdns/pod-to-world/http-to-one.one.one.one-0: cilium-test/client-6b4b857d98-b2mts (10.0.12.196) -> one.one.one.one-http (one.one.one.one:80)
  ❌ to-fqdns/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-646b88fb9b-xsb7z (10.0.12.156) -> one.one.one.one-http (one.one.one.one:80)
connectivity test failed: 4 tests failed

Please note that the same test works fine when disabling the following values of the Helm chart:

envoy:
  enabled: true

The results when disabling Envoy as DaemonSet:

✅ All 42 tests (284 actions) successful, 13 tests skipped, 0 scenarios skipped.

Cilium Version

cilium version
cilium-cli: v0.15.7 compiled with go1.21.0 on linux/amd64
cilium image (default): v1.14.1
cilium image (stable): v1.14.1
cilium image (running): 1.14.1

Kernel Version

Latest bottlerocket AMI

Kubernetes Version

v1.27.4

Sysdump

cilium-sysdump-20230910-102356.zip

Relevant log output

No response

Anything else?

The whole code used is here

Code of Conduct

• I agree to follow this project's Code of Conduct

[aanm]

@Smana can you try the test again? Since the connectivity was to 1.1.1.1 it might have failed because it was making connections to the outside world.

[Smana]

@aanm Of course, do I have to change anything? upgrade the cilium CLI version?

[aanm]

@Smana no, a simple re-run should be enough.

[SmaineTF1]

I just run the command again and I still get the same errors. I don't understand how it would work without changing anything?
I'm gonna double check again without running envoy as daemonset

[aanm]

@SmaineTF1 Just because we have had CI flakes for connectivity tests done to the outside world.

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has not seen any activity since it was marked stale.
Closing.