Nodeport service is broken with proxy-mode=ipvs

[evgkrsk]

I observe constant timeouts while containerd trying to access 127.0.0.1:30021 local mirror:

root@myhostname:~# journalctl -r -u containerd |grep 30021 |head
Nov 01 00:29:36 myhostname containerd[2413588]: time="2023-11-01T00:29:36.775166395Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/rtl/marketing-promocode/be-1301/manifests/49ad4e18?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:33 myhostname containerd[2413588]: time="2023-11-01T00:29:33.774039526Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/mkk/loan/feature-mkk-3870/manifests/bf6460d0?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:26 myhostname containerd[2413588]: time="2023-11-01T00:29:26.775053471Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/opendatacollecting/eparser/odc-2259/manifests/ee2d26a6?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:24 myhostname containerd[2413588]: time="2023-11-01T00:29:24.843621330Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/inventory/api/supply-return-readmodel/inv-2632/manifests/ddd7f161?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:05 myhostname containerd[2413588]: time="2023-11-01T00:29:05.787353142Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/wms/svc/logistics-megasort-facade/wms-41662/manifests/ca271126?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:28:59 myhostname containerd[2413588]: time="2023-11-01T00:28:59.053739919Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/eea/platform/exteca.platform.orchestrator.sublots/master/manifests/9c2bfe17?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:28:26 myhostname containerd[2413588]: time="2023-11-01T00:28:26.772423981Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/eea/platform/exteca.platform.orchestrator.sublots/master/manifests/9c2bfe17?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:28:01 myhostname containerd[2413588]: time="2023-11-01T00:28:01.977061019Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/lsplt/trip-container-service/trip-container-business-layer/master/manifests/f07adc9d?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:27:49 myhostname containerd[2413588]: time="2023-11-01T00:27:49.779983205Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/pricing/services/pricing-comments/f-ppric-3865/manifests/ed7e4e82?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:27:07 myhostname containerd[2413588]: time="2023-11-01T00:27:07.147305757Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/kdp/api/onec-gateway-api/master/manifests/c495844d?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"

Seems like access from hostnetwork to 127.0.0.1:nodeport working only with proxy-mode=iptables (see: kubernetes/kubernetes#111840). As result, deploying spegel in cluster with ipvs make image pull times much worse (1s->60s).

How can we handle this issue without falling back to iptables?

[evgkrsk]

I manage to workaround problem using external IP instead of 127.0.0.1:30021 in mirror configuration, seems to work fine (I use in-house method for mirror configuration, not spegel's).

[phillebaba]

Adding an option to use the Node IP rather than localhost in the Helm chart should not be an issue. I guess the question is if this change should be made for everyone or if it should be behind an option for ipvs users.

[evgkrsk]

It may be behind option, but turned on by default to simplify installation in different envs.

[phillebaba]

I would prefer it to be an opt in option in that case.