cri-o needs cni workaround #1804

mike-nguyen · 2018-09-19T16:26:37Z

Description
Port forwarding from the container to the host doesn't seem to be working. I need to run with hostNetwork: true to workaround it.

I am using kubelet to deploy a static nginx pod with containerPort:80 and hostPort:80. When kubelet is using docker, I can curl http://localhost:80 and I get the nginx default page. When I switch kubelet to run with crio, I can see the container running but curl http://localhost:80 does not respond. This is a similar issue I was seeing with exposing ports using podman.

cri-o might need a version of containers/podman#1431
mheon mentioned:

You can probably vendor libpod's pkg/firewall and then modify your netns setup code to run it though you might need to store more information than you do right now, libpod already stores the whole CNI response CRI-O does not

Steps to reproduce the issue:
1.

cat << EOF > /etc/kubernetes/manifests/static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
 name: static-web
 labels:
   role: myrole
spec:
 containers:
   - name: web
     image: docker.io/nginx
     ports:
       - name: web
         containerPort: 80
         hostPort: 80
EOF

cat << EOF > /etc/kubernetes/kubeletconfig
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
clusterDomain: kube.local
clusterDNS: [10.254.0.10]
cgroupDriver: systemd
staticPodPath: /etc/kubernetes/manifests/
authorization:
  mode: AlwaysAllow
authentication:
  anonymous:
    enabled: true
  webhook:
    enabled: false
EOF

start kubelet
/usr/bin/hyperkube kubelet --config /etc/kubernetes/kubeletconfig --container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m
4
curl http://localhost:80

Describe the results you received:
Cannot access the nginx container through the exposed host port

Describe the results you expected:
Able to access the nginx container through the exposed host port

Additional information you deem important (e.g. issue happens only occasionally):
Always happens

Output of crio --version:

crio version 1.11.4
cri-o-1.11.4-2.rhaos3.11.gite0c89d8.el7_5.x86_64

Additional Environment Information
RHCOS 4.0.5796 in AWS

The text was updated successfully, but these errors were encountered:

mike-nguyen · 2018-09-19T16:28:01Z

/cc @mrunalp @ashcrow @mheon

giuseppe · 2018-10-02T17:15:28Z

could you verify if it works when you try to access another host IP that is not localhost? For example, it works when I use the local IP 192.168.125.240 but it hangs on 127.0.0.1.

Using the firewall plugin from containernetworking/plugins#75 doesn't seem to make a difference.

/cc @dcbw

mike-nguyen · 2018-10-02T17:36:48Z

@giuseppe I tested on AWS and was able to get to my webserver through the AWS private IP address. So it only hangs on localhost/127.0.0.1 like you said

dcbw · 2018-10-02T18:12:27Z

@mike-nguyen can you run 'iptables-save' on the node and post the results somewhere?

mike-nguyen · 2018-10-02T18:15:48Z

@dcbw https://paste.fedoraproject.org/paste/vJ6Ney7VbkNWojiTj80~rg

dcbw · 2018-10-02T18:28:54Z

@mike-nguyen can you "tcpdump -vvvne cni0" and then do the curl localhost:80?

mike-nguyen · 2018-10-02T19:23:33Z

@dcbw tcpdump -vvvne -i cn0 gives me no output when I curl localhost:80

dcbw · 2018-10-03T14:58:26Z

@mike-nguyen Ok, so clearly the traffic isn't even getting to cni0. That helps narrow things down.

giuseppe · 2018-10-04T16:48:04Z

@dcbw I got it to work with the following changes (tested manually):

@@ -31,9 +31,10 @@
 -A PREROUTING -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS
 -A OUTPUT -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS
 -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
+-A POSTROUTING -d 10.88.0.0/16 -m comment --comment "name: \"crio-bridge\" id: \"bf266838ac8d1fc99c6efee82fbe9333d55ae3eecec3248e9e2f568561d1ad8d\"" -j CNI-08a451cf4b6dd952bc70028c
 -A POSTROUTING -s 10.88.0.0/16 -m comment --comment "name: \"crio-bridge\" id: \"bf266838ac8d1fc99c6efee82fbe9333d55ae3eecec3248e9e2f568561d1ad8d\"" -j CNI-08a451cf4b6dd952bc70028c
 -A POSTROUTING -s 127.0.0.0/8 -o lo -m comment --comment "SNAT for localhost access to hostports" -j MASQUERADE
--A CNI-08a451cf4b6dd952bc70028c -d 10.88.0.0/16 -m comment --comment "name: \"crio-bridge\" id: \"bf266838ac8d1fc99c6efee82fbe9333d55ae3eecec3248e9e2f568561d1ad8d\"" -j ACCEPT
+-A CNI-08a451cf4b6dd952bc70028c -d 10.88.0.0/16 -m comment --comment "name: \"crio-bridge\" id: \"bf266838ac8d1fc99c6efee82fbe9333d55ae3eecec3248e9e2f568561d1ad8d\"" -j MASQUERADE
 -A CNI-08a451cf4b6dd952bc70028c ! -d 224.0.0.0/4 -m comment --comment "name: \"crio-bridge\" id: \"bf266838ac8d1fc99c6efee82fbe9333d55ae3eecec3248e9e2f568561d1ad8d\"" -j MASQUERADE
 -A KUBE-HOSTPORTS -p tcp -m comment --comment "k8s_nginx-rhel7_default_6cce1e24d665240f8d2e114edbf52189_3_ hostport 8081" -m tcp --dport 8081 -j KUBE-HP-WWG7ZHMF34UGM273
 -A KUBE-HP-WWG7ZHMF34UGM273 -s 10.88.0.59/32 -m comment --comment "k8s_nginx-rhel7_default_6cce1e24d665240f8d2e114edbf52189_3_ hostport 8081" -j KUBE-MARK-MASQ

giuseppe · 2018-10-18T08:47:07Z

@dcbw does the change look fine?

mike-nguyen · 2018-11-01T13:04:32Z

@dcbw -- looks like giuseppe has something that may work. What do you think?

ashcrow · 2018-11-01T15:12:11Z

If we can get this looked at soon I'd appreciate it. This work blocks a card @mike-nguyen is trying to close out.

mrunalp · 2019-01-05T17:18:33Z

@dcbw wdyt of @giuseppe changes?

dcbw · 2019-01-15T19:50:42Z

@squeed

squeed · 2019-01-16T17:44:51Z

You need to masquerade, but only when the source address is 127.0.0.1. Check out https://github.com/containernetworking/plugins/tree/master/plugins/meta/portmap#snat-masquerade

mrunalp · 2019-03-08T05:23:45Z

@squeed how do we get this fixed in the CNI plugins?

squeed · 2019-03-11T16:53:04Z

The CNI plugins already handle this; what version and what CNI configuration are you using?

mike-nguyen · 2019-04-25T21:17:18Z

The CNI plugins already handle this; what version and what CNI configuration are you using?

@squeed Sorry we swapped jira boards and the card got lost in the migration. Let me know if you need anymore information.

$ rpm -q containernetworking-plugins 
containernetworking-plugins-0.7.4-3.git9ebe139.module+el8.0.0+2958+4e823551.x86_64

These are the configs in /etc/cni/net.d:

100-crio-bridge.conf

{
    "cniVersion": "0.3.0",
    "name": "crio-bridge",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ]
    }
}

200-loopback.conf

{
    "cniVersion": "0.3.0",
    "type": "loopback"
}

87-podman-bridge.conflist

{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "subnet": "10.88.0.0/16",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ]
        }
      },
      {
        "type": "portmap",
        "capabilities": {
          "portMappings": true
        }
      }
    ]
}

bkmeneguello · 2019-10-18T19:08:46Z

I'm seeing this error with:

Centos 7
containernetworking-plugins-0.8.1-2.el7.centos.x86_64
cri-o-1.15.2-1.el7.x86_64
kubelet-1.16.2-0.x86_64
quay.io/coreos/flannel:v0.11.0-amd64

Any solution available?

prasenforu · 2020-04-02T18:00:54Z

so far any workaround ?

haircommander · 2020-04-29T18:42:20Z

@squeed @dcbw @giuseppe any further thoughts here?

amartinunowhy · 2021-12-03T13:50:23Z

Hello. I'm not sure I'm having the same issue, but looking at what podman does and the differences, I found that by adding the following ip-tables rule, it works :

 -A KUBE-HP-GSFDA2UGA5JYS6WS -s 10.85.1.12/32 -m comment --comment "k8s_nginx-sandbox_default_hdishd83djaidwnduwk28bcsb_1_ hostport 8089" -j KUBE-MARK-MASQ
+ -A KUBE-HP-GSFDA2UGA5JYS6WS -s 127.0.0.1/32 -m comment --comment "k8s_nginx-sandbox_default_hdishd83djaidwnduwk28bcsb_1_ hostport 8089" -j KUBE-MARK-MASQ

My understanding is that KUBE-MARK-MASQ adds a mark:

-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000

And that the mark allows the following rules to be run:

-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully

and in particular the last one. I don't really understand what it does but it may be because as 127.0.0.1 exists in the container, without masquerading the source IP the response packets would never go out of it.

Hope it helps.

amartinunowhy · 2021-12-03T17:30:03Z

Hi again. Looking at the source code, I would add the following code here: https://github.com/cri-o/cri-o/blob/main/internal/hostport/hostport_manager.go#L151

// SNAT if the traffic comes from the host through localhost (127.0.0.1)
if pm.HostIP == "" || pm.HostIP == "0.0.0.0" || pm.HostIP == "::"  || pm.HostIP == "127.0.0.1" {
	writeLine(natRules, "-A", string(chain),
		"-m", "comment", "--comment", fmt.Sprintf(`"%s hostport %d"`, podFullName, pm.HostPort),
		"-s 127.0.0.1",
		"-j", string(iptablesproxy.KubeMarkMasqChain))
}

I'm sorry but I am unable to test myself.

haircommander · 2021-12-03T19:25:38Z

thanks for looking into it! would you be interested in opening a PR @amartinunowhy ? cc @aojea

aojea · 2021-12-04T10:20:54Z

/assign
I'm on vacation until end of next week, but I'll take a look at this, I vaguely remember kubelet is also blocking some 127.0.0.1 traffic due to a CVE, I want to check first if that can impact this

github-actions · 2022-06-30T08:37:59Z

A friendly reminder that this issue had no activity for 30 days.

aojea · 2022-06-30T09:00:20Z

/unassign

I'm sorry but I didn't have time for this, squeed reference to the cni portmap plugin is right

https://github.com/containernetworking/plugins/blob/9c59728d39b729558bb712c79e2c80acc0864191/plugins/meta/portmap/portmap.go#L77-L99

and seems to match the solutions proposed

#1804 (comment)

I'm happy to act as a reviewer, but I will not able to send PRs for this

haircommander · 2023-05-24T16:13:15Z

it doesn't seem like this is a very high prio bug (there aren't that many people looking for a fix). It also doesn't seem we have the bandwidth to fix the bug. If someone else sees the bug and wants it fixed, please reopen. Otherwise, I'm leaving it closed.

mike-nguyen mentioned this issue Sep 19, 2018

kola/tests: add kubernetes static pod test coreos/mantle#910

Open

afbjorklund mentioned this issue Mar 10, 2020

update registry addon documentation kubernetes/minikube#6982

Open

openshift-ci bot assigned aojea Dec 4, 2021

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 30, 2022

openshift-ci bot unassigned aojea Jun 30, 2022

haircommander closed this as completed May 24, 2023

jperville mentioned this issue Dec 13, 2023

How to configure the proxy bind address ? enix/kube-image-keeper#229

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cri-o needs cni workaround #1804

cri-o needs cni workaround #1804

mike-nguyen commented Sep 19, 2018 •

edited

mike-nguyen commented Sep 19, 2018

giuseppe commented Oct 2, 2018

mike-nguyen commented Oct 2, 2018

dcbw commented Oct 2, 2018

mike-nguyen commented Oct 2, 2018 •

edited

dcbw commented Oct 2, 2018

mike-nguyen commented Oct 2, 2018

dcbw commented Oct 3, 2018 •

edited

giuseppe commented Oct 4, 2018

giuseppe commented Oct 18, 2018

mike-nguyen commented Nov 1, 2018

ashcrow commented Nov 1, 2018

mrunalp commented Jan 5, 2019

dcbw commented Jan 15, 2019

squeed commented Jan 16, 2019

mrunalp commented Mar 8, 2019

squeed commented Mar 11, 2019

mike-nguyen commented Apr 25, 2019

bkmeneguello commented Oct 18, 2019

prasenforu commented Apr 2, 2020

haircommander commented Apr 29, 2020

amartinunowhy commented Dec 3, 2021 •

edited

amartinunowhy commented Dec 3, 2021

haircommander commented Dec 3, 2021

aojea commented Dec 4, 2021

github-actions bot commented Jun 30, 2022

aojea commented Jun 30, 2022

haircommander commented May 24, 2023

cri-o needs cni workaround #1804

cri-o needs cni workaround #1804

Comments

mike-nguyen commented Sep 19, 2018 • edited

mike-nguyen commented Sep 19, 2018

giuseppe commented Oct 2, 2018

mike-nguyen commented Oct 2, 2018

dcbw commented Oct 2, 2018

mike-nguyen commented Oct 2, 2018 • edited

dcbw commented Oct 2, 2018

mike-nguyen commented Oct 2, 2018

dcbw commented Oct 3, 2018 • edited

giuseppe commented Oct 4, 2018

giuseppe commented Oct 18, 2018

mike-nguyen commented Nov 1, 2018

ashcrow commented Nov 1, 2018

mrunalp commented Jan 5, 2019

dcbw commented Jan 15, 2019

squeed commented Jan 16, 2019

mrunalp commented Mar 8, 2019

squeed commented Mar 11, 2019

mike-nguyen commented Apr 25, 2019

bkmeneguello commented Oct 18, 2019

prasenforu commented Apr 2, 2020

haircommander commented Apr 29, 2020

amartinunowhy commented Dec 3, 2021 • edited

amartinunowhy commented Dec 3, 2021

haircommander commented Dec 3, 2021

aojea commented Dec 4, 2021

github-actions bot commented Jun 30, 2022

aojea commented Jun 30, 2022

haircommander commented May 24, 2023

mike-nguyen commented Sep 19, 2018 •

edited

mike-nguyen commented Oct 2, 2018 •

edited

dcbw commented Oct 3, 2018 •

edited

amartinunowhy commented Dec 3, 2021 •

edited