No access to service-cidr with netstack

[rlex]

Description

k3s v1.22.3+k3s1
containerd://1.5.7-k3s2
cilium 1.10.0 (if that matters). Kube-proxy replacement enabled with eBPF dataplane.

gvisor installed and working.
However, DNS resolution fails, because gvisor without host-network fails to access 10.43.0.0/16 network which is assigned service-cidr. Network otherwise works (i was able to curl https://1.1.1.1, for example).
Works fine with host-network, but i assume this is overkill for just dns access?

Happy to help with debugging if you can point me to right direction.

Steps to reproduce

containerd config:

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "rancher/mirrored-pause:3.1"
[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins.cri.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc-hostnetwork]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"
[plugins.cri.containerd.runtimes.runsc-hostnetwork.options]
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc-hostnetwork.toml"

(I have two runtimeclasses with hostnetwork and without)

/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml is empty atm

runsc version

runsc version release-20211129.0
spec: 1.0.2

docker version (if using docker)

No response

uname

Linux node-4 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

kubectl (if using Kubernetes)

No response

repo state (if built from source)

No response

runsc debug logs (if available)

No response

[rlex]

routing table on gvisor with netstack (using sample nginx pod):

root@gvisor:/# cat /proc/net/route 
Iface   Destination     Gateway Flags   RefCnt  Use     Metric  Mask    MTU     Window  IRTT                                                         
eth0    F802000A        00000000        0001    0       0       0       FFFFFFFF        0       0       0                                                                               
eth0    00000000        F802000A        0003    0       0       0       00000000        0       0       0 

root@gvisor:/# route  
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.2.248      0.0.0.0         255.255.255.255 U     0      0        0 eth0
default         10.0.2.248      0.0.0.0         UG    0      0        0 eth0

with hostnetwork:

root@gvisor-hostnetwork:/# cat /proc/net/route 
Iface   Destination     Gateway Flags   RefCnt  Use     Metric  Mask    MTU     Window  IRTT                                                         
eth0    00000000        EC06000A        0003    0       0       0       00000000        0       0       0                                                                               
eth0    EC06000A        00000000        0001    0       0       0       FFFFFFFF        0       0       0                                                                               
eth0    3206000A        00000000        0001    0       0       0       FFFFFFFF        0       0       0 

root@gvisor-hostnetwork:/# route         
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.6.236      0.0.0.0         UG    0      0        0 eth0
10.0.6.236      0.0.0.0         255.255.255.255 U     0      0        0 eth0
10.0.6.50       0.0.0.0         255.255.255.255 U     0      0        0 eth0

[hbhasker]

Do you have a minimal repro? I am not sure I follow what DNS and the address in this comment have to do with each other
.
"However, DNS resolution fails, because gvisor without host-network fails to access 10.43.0.0/16 network which is assigned"

[rlex]

DNS (coredns in my example) is deployed as an in-cluster service with ip 10.43.0.10, which is part of service-cidr (k3s default is 10.43.0.0/16)
Other IPs in this range can't be reached with default gvisor config (non-hostnetwork one), for example default "kubernetes" service located on 10.43.0.1 which is kubernetes-api.
As soon as i enable host-network in runsc config, both DNS and kubernetes API starts working.

[hbhasker]

Sorry I dropped the ball on this. Got busy with other things. It would be great if you could setup a demo cluster/project and give me access so that I can see what's going on.

[rlex]

@hbhasker yeah it's possible, i can spin up some small vm somewhere. How can i contact you, slack / etc?

[hbhasker]

Please drop me a note at bhaskerh at Google dot com.

[github-actions]

A friendly reminder that this issue had no activity for 120 days.

[github-actions]

This issue has been closed due to lack of activity.