CI: Conformance Runtime (ci-runtime): Runtime (agent): DNS proxy policy works if Cilium stops

[gandro]

CI failure

/home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:515
07:49:15 STEP: Setting up policy: /root/go/src/github.com/cilium/cilium/test/policy_644568a1.json
07:49:16 STEP: Setting up policy: /root/go/src/github.com/cilium/cilium/test/policy_a304e84d.json
07:49:17 STEP: Curl from "app1" to "http://world1.cilium.test/"
07:49:17 STEP: Curl from "app1" to "http://world1.outside.test/" should fail
07:49:22 STEP: Dumping IP cache before Cilium is stopped
Local scope identities in IP cache before Cilium restart: map[172.18.0.2/32:16777223 192.168.1.42/32:16777222 fe80::/10:16777219 fe80::1/128:16777220 fe80::2/128:16777221]
07:49:23 STEP: Stopping Cilium
cilium.test selectors before restart: 16777218
16777223

07:49:23 STEP: Testing connectivity from "app1" to the IP "172.18.0.2" without DNS request
07:49:23 STEP: Curl from "app1" to "http://world1.cilium.test/" with Cilium down
07:49:28 STEP: Testing that invalid traffic is still block when Cilium is down%!(EXTRA string=app1, string=172.18.0.7)
07:49:33 STEP: Starting Cilium again
07:49:33 STEP: Restarting Cilium
07:49:45 STEP: Dumping IP cache after Cilium is restarted
Local scope identities in IP cache after Cilium restart: map[172.18.0.2/32:16777223 192.168.1.42/32:16777222 fe80::/10:16777219 fe80::1/128:16777220 fe80::2/128:16777221]
07:49:45 STEP: Setting up policy: /root/go/src/github.com/cilium/cilium/test/policy_c64253b9.json
cilium.test selectors after restart: 16777218
16777224

FAIL: Expected
    <string>: 16777218\n16777224\n
to equal
    <string>: 16777218\n16777223\n
=== Test Finished at 2024-06-19T07:49:45Z====

• Failure [41.399 seconds]
RuntimeAgentFQDNPolicies
/home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:461
  DNS proxy policy works if Cilium stops [It]
  /home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:515

  Expected
      <string>: 16777218\n16777224\n
  to equal
      <string>: 16777218\n16777223\n

  /home/runner/work/cilium/cilium/test/runtime/fqdn.go:1139

• https://github.com/cilium/cilium/actions/runs/9578069113/job/26407602863
• Occurred in toFQDNs: Add documention and metrics for fqdn identities #33237 which shouldn't have affected regular operations

logs_25069779201.zip

[gandro]

Also found in an earlier run https://github.com/cilium/cilium/actions/runs/9548860479/job/26317322574

[gandro]

I think the following is happening right now:

1. We do a lookup for world1.cilium.io, it returns exactly 1 IP per IP family. We allocate one FQDN identity (fqdn:world1.cilium.io,reserved:world-ipvN) per family
2. We shut down Cilium, where it didn't manage to checkpoint the latest FQDN identities
3. Cilium comes up again, it sees the IP(s) in IPCache and withholds the numeric identity. Because it has no labels and the identity is unique to the IP, it adds a cidr label. The IP is therefore added as a "restored" CIDR idenity to the new cache
4. FQDN policy is re-imported (after endpoint regeneration!), because it now adds a fqdn:world1.cilium.io label to the IP, we allocate a new numeric identity for the IP
5. The test check numeric identities and is unhappy about the fact that identities changed

[gandro]

The "good" news is that this identity change shouldn't actually cause drops, because it's happening after restoration and this is mainly a false positive in terms of test failure. Still, we want to fix either the test or the check-pointing (or both)