CI: external workloads workflow consistently fails in "Verify DNS on VM" step

[tklauser]

The external workloads workflow seems to be failing consistently:

https://github.com/cilium/cilium-cli/actions/runs/6622170504

It looks like nslookup is timing out trying to reach the DNS server:

;; connection timed out; no servers could be reached

This seems to have started around 2023-10-21.

• First scheduled run that failed: https://github.com/cilium/cilium-cli/actions/runs/6593096759
• Last scheduled run that passed: https://github.com/cilium/cilium-cli/actions/runs/6590038993

[nbusseneau]

External workfloads on main Cilium repo is working, perhaps something was changed in there that should have been changed on CLI as well but did not?

[michi-covalent]

this might be relevant. from the sysdump in https://github.com/cilium/cilium-cli/actions/runs/6725580157 cilium-sysdump-out.zip.zip

% grep 'level=error' ./logs-clustermesh-apiserver-694765f445-9z5d6-apiserver-20231101-221630.log
2023-11-01T22:13:59.507218421Z level=error msg="CEW: Invalid identity 1 in &{{cilium-cilium-cli-6725580157-classic-vm1 cilium-cilium-cli-6725580157-classic-vm1 [{InternalIP 10.168.0.5} {InternalIP fc00::10ca:1} {CiliumInternalIP 10.192.1.2} {CiliumInternalIP f00d::a05:0:0:9f6f}] 10.192.1.0/30 [] f00d::a05:0:0:0/96 [] <nil> <nil> <nil> <nil> 0 local 0 map[io.cilium.k8s.policy.cluster:cilium-cilium-cli-6725580157-classic-vm1 io.kubernetes.pod.name:cilium-cilium-cli-6725580157-classic-vm1 io.kubernetes.pod.namespace:default] map[] 1 }}" subsys=clustermesh-apiserver

[giorio94]

this might be relevant. from the sysdump in https://github.com/cilium/cilium-cli/actions/runs/6725580157 cilium-sysdump-out.zip.zip

% grep 'level=error' ./logs-clustermesh-apiserver-694765f445-9z5d6-apiserver-20231101-221630.log
2023-11-01T22:13:59.507218421Z level=error msg="CEW: Invalid identity 1 in &{{cilium-cilium-cli-6725580157-classic-vm1 cilium-cilium-cli-6725580157-classic-vm1 [{InternalIP 10.168.0.5} {InternalIP fc00::10ca:1} {CiliumInternalIP 10.192.1.2} {CiliumInternalIP f00d::a05:0:0:9f6f}] 10.192.1.0/30 [] f00d::a05:0:0:0/96 [] <nil> <nil> <nil> <nil> 0 local 0 map[io.cilium.k8s.policy.cluster:cilium-cilium-cli-6725580157-classic-vm1 io.kubernetes.pod.name:cilium-cilium-cli-6725580157-classic-vm1 io.kubernetes.pod.namespace:default] map[] 1 }}" subsys=clustermesh-apiserver

I did a few tests, and this error message is reported also when using earlier versions (both v1.13.x and v1.14.2), so I'd say it should not be strictly related. That said, the cause is [1] (which overrides the identity created by the clustermesh-apiserver with the default one of nodes), but I'm not 100% sure if it is a real bug or not (I guess it might impact policy enforcement towards external workloads).

But I'm still confused why we are seeing this error only in the cilium/cilium-cli workflows. I've tried comparing the agent parameters with those on cilium/cilium, and there seems to be no difference. I've also tried reproducing this locally, without any luck.

[1]: https://github.com/cilium/cilium/blob/4f08f6cc7029789cd6aae9f6d4e70ec9613c8442/pkg/nodediscovery/nodediscovery.go#L195-L218

[giorio94]

Had a fresh look again, and this looks incredibly suspicious.

v1.14.2:

$ grep routing-mode cilium-configmap-* 
routing-mode: tunnel

v1.14.3:

$ grep routing-mode cilium-configmap-* 
routing-mode: native

The behavioral change has been introduced in cilium/cilium@381e4ec15334.

$ helm template cilium install/kubernetes/cilium -n kube-system -s templates/cilium-configmap.yaml --set gke.enabled=true --set routingMode=tunnel | grep routing-mode

# Before:
  routing-mode: "native"
# After:
  routing-mode: "native"
  routing-mode: "tunnel"

Still unclear to me why it is not happening in cilium/cilium though.

Edit: this is the likely reason why we didn't hit it in cilium/cilium:

CILIUM_INSTALL_DEFAULTS="--cluster-name=${{ env.clusterName }} \
            --datapath-mode=tunnel \ <----
            --chart-directory=install/kubernetes/cilium \
            --helm-set=image.repository=quay.io/${{ env.QUAY_ORGANIZATION_DEV }}/cilium-ci \
            ...