Requests to local spegel instance are being detected as external with cilium

[Preisschild]

Hi

I tried Spegel recently and found that when a node pulls an image (etcd in this case), the Spegel instance on the node detects it as an external request:

{"level":"info","ts":1697181744.1945643,"caller":"registry/registry.go:174","msg":"handling mirror request from external node","path":"/v2/etcd-development/etcd/blobs/sha256:47ba7aff063ffd3883d25edc17bc0a92a4b76d48bcc89e8a21a149db234576bd","ip":"10.65.10.213"}

The IP (10.65.10.213) is the one assigned to the cilium_host interface.

I'm not sure how to fix this. Is someone here using Spegel with Cilium?

[phillebaba]

The logic to determine an external request is very simple, maybe a bit too simple.

https://github.com/XenitAB/spegel/blob/4a190529ade0eabbfcdc107c8173ef39b6c2f3b8/internal/registry/registry.go#L290-L292

My guess is that Cilium implements host networking differently which results in a different host header.

We need to setup a test cluster running Cilium to determine what this host header is.

[Preisschild]

The host header seems to be an IP (10.65.10.213) specifically owned by the node-specific cilium-agent (over the cilium_host network interface. It is located within the node-specific podCIDR 10.65.10.0/24.

I already found that function, but I'm not entirely sure how to handle this.

[phillebaba]

I might have some time now to work on this. The logic to determine if the request is external is not ideal to be honest. Might just replace it all together if I am able to.

[phillebaba]

So the issue is not that the request is not being detected, but that the way that host ports on Cilium are implemented is a bit different. #258 has a different but a similar issue. The best solution is to change the host port to a node port service with a local internal traffic policy. Then we change the mirror configuration to use the node IP instead of localhost to reach Spegel. That way we solve both issues and also move away from host port mapping which seems to have varying implementations in all CNIs.

This will however require Kubernetes 1.26, which may be a problem for some. I might just add an option to use the old host port for those who are not running 1.26 yet.

[phillebaba]

As an update I do not think I will be able to solve this with a node port service due to the different implementations.

The issue is caused by some weird implementations of host port mapping in Cilium which has been described in an issue.

cilium/cilium#12116

I do not have enough experience with Cilium to describe the required configuration to make port mapping work properly. It would be great to get some help if someone does know and can help with documentation.

[joemiller]

I'm noticing this in my logs too (and I use cilium), but I am not sure what the impact is? Looking through the code I suspect maybe it only matters for metrics?

[Preisschild]

I finally found the problem why it was not working when i saw this line in the cilium-agent logs:

The requested hostPort 30020 is colliding with the configured NodePort range [30000, 32767]. Ignoring.

After setting service.registry.hostPort to a port outside this range everything worked.

I'm closing this issue now, as there don't seem to be any problems on Cilium