"No route to host" while trying to curl to the container connected to multiple networks using cni when containernetworking-plugins version is 1.0.0 and onwards

[yklin2022]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When a container is publishing ports with <host_ip>:port, and is connected to multiple (non-internal) bridge network, using the network backend cni + dnsname plugin, trying to curl the container with <host_ip>:port shows no route to host, if the containernetworking-plugins version >= 1.0.0 (0.9.1 is working fine)

Steps to reproduce the issue:

Note: containernetworking-plugins version has to be >= 1.0.0

1. podman network create --driver=bridge --subnet=192.168.230.0/24 network_main
   podman network create --driver=bridge --subnet=192.168.228.0/24 network_sub

2. podman run -d --network=network_main -p 10.85.41.219:9000:8080 --name nginx docker.io/bitnami/nginx:latest

3. podman network connect network_sub nginx

Describe the results you received:
curl: (7) Failed to connect to 10.85.41.219 port 9000: No route to host

Describe the results you expected:
curl 10.85.41.219:9000 should work as well, since it was working when I use containernetworking-plugins version 0.9.1

Additional information you deem important (e.g. issue happens only occasionally):
I tried containernetworking-plugins version 1.0.0, 1.0.1 and 1.1.1, none of them works; I can only fall back to version 0.9.1

I reported another issue with same symptoms using network backend netavark, for that one, it is irrelevant to containernetworking-plugins version (I think it makes sense since containernetworking-plugins is for cni if I understand it correctly)

Output of podman version:

Client:       Podman Engine
Version:      4.0.0
API Version:  4.0.0
Go Version:   go1.17.7

Built:      Mon Feb 21 13:35:55 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: ae467a0c8001179d4d0adf4ada381108a893d7ec'
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: xyz
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-305.40.2.el8_4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 58189189120
  memTotal: 66800742400
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.0.0-74.rc95.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.13
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 68719472640
  swapTotal: 68719472640
  uptime: 15m 48.44s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 9
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.0
  Built: 1645479355
  BuiltTime: Mon Feb 21 13:35:55 2022
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.0.0-3.module+el8.6.0+14305+6b14f34e.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
I'm running Podman as root
Firewalld is disabled on the host

[mheon]

Seems like this is probably a CNI bug. Can you provide your current firewall rules when the container is running and symptoms are observed (iptables -nvL and iptables -t nat -nvL).

[yklin2022]

For containernetworing-plugins version 0.9.1: (curl to container IS working fine)
Output of iptables -nvL:

Chain INPUT (policy ACCEPT 5975 packets, 37M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin rules */

Chain OUTPUT (policy ACCEPT 1939 packets, 186K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain CNI-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-ADMIN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin admin overrides */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.230.2        ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      *       192.168.230.2        0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.228.2        ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      *       192.168.228.2        0.0.0.0/0           

Chain CNI-ADMIN (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Output of iptables -t nat -nvL:

Chain PREROUTING (policy ACCEPT 151 packets, 23891 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 129 packets, 21303 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 22 packets, 1640 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   22  1640 CNI-HOSTPORT-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd requiring masquerade */
    0     0 CNI-c65b0350176d318d547eec6a  all  --  *      *       192.168.230.2        0.0.0.0/0            /* name: "network_main" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */
    0     0 CNI-53e1f04f3597bcf67a4ba8dc  all  --  *      *       192.168.228.2        0.0.0.0/0            /* name: "network_sub" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */

Chain OUTPUT (policy ACCEPT 20 packets, 1520 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2   120 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain CNI-c65b0350176d318d547eec6a (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.230.0/24     /* name: "network_main" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4          /* name: "network_main" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */

Chain CNI-HOSTPORT-SETMARK (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd masquerade mark */ MARK or 0x2000

Chain CNI-HOSTPORT-MASQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

Chain CNI-HOSTPORT-DNAT (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   120 CNI-DN-c65b0350176d318d547ee  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* dnat name: "network_main" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */ multiport dports 9000
    0     0 CNI-DN-53e1f04f3597bcf67a4ba  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* dnat name: "network_sub" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */ multiport dports 9000

Chain CNI-DN-c65b0350176d318d547ee (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       192.168.230.0/24     10.85.41.219         tcp dpt:9000
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            10.85.41.219         tcp dpt:9000
    2   120 DNAT       tcp  --  *      *       0.0.0.0/0            10.85.41.219         tcp dpt:9000 to:192.168.230.2:8080

Chain CNI-53e1f04f3597bcf67a4ba8dc (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.228.0/24     /* name: "network_sub" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4          /* name: "network_sub" id: "2c6d75e477b3bd275e5b172643fd6722d3fd2d4c3f54724eebe07f5540a4abf8" */

Chain CNI-DN-53e1f04f3597bcf67a4ba (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       192.168.228.0/24     10.85.41.219         tcp dpt:9000
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            10.85.41.219         tcp dpt:9000
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            10.85.41.219         tcp dpt:9000 to:192.168.228.2:8080

For containernetworing-plugins version 1.1.1: (curl to container NOT working, show "No route to host")
Output of iptables -nvL:

Chain INPUT (policy ACCEPT 9680 packets, 58M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin rules */

Chain OUTPUT (policy ACCEPT 4040 packets, 389K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain CNI-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-ADMIN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin admin overrides */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.230.3        ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      *       192.168.230.3        0.0.0.0/0           
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.228.3        ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      *       192.168.228.3        0.0.0.0/0           

Chain CNI-ADMIN (1 references)
 pkts bytes target     prot opt in     out     source               destination   

Output of iptables -t nat -nvL:

Chain PREROUTING (policy ACCEPT 502 packets, 81058 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 436 packets, 73294 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 81 packets, 6012 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   81  6012 CNI-HOSTPORT-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd requiring masquerade */
    0     0 CNI-eabfde9de4e0fe8c6c1f933d  all  --  *      *       192.168.230.3        0.0.0.0/0            /* name: "network_main" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */
    0     0 CNI-db249f3a7e41a3fb137d5d42  all  --  *      *       192.168.228.3        0.0.0.0/0            /* name: "network_sub" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */

Chain OUTPUT (policy ACCEPT 76 packets, 5712 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    5   300 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain CNI-HOSTPORT-SETMARK (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd masquerade mark */ MARK or 0x2000

Chain CNI-HOSTPORT-MASQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

Chain CNI-HOSTPORT-DNAT (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    3   180 CNI-DN-eabfde9de4e0fe8c6c1f9  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* dnat name: "network_main" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */ multiport dports 9000
    0     0 CNI-DN-db249f3a7e41a3fb137d5  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* dnat name: "network_sub" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */ multiport dports 9000

Chain CNI-eabfde9de4e0fe8c6c1f933d (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.230.0/24     /* name: "network_main" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4          /* name: "network_main" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */

Chain CNI-DN-eabfde9de4e0fe8c6c1f9 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       192.168.230.0/24     10.85.41.219         tcp dpt:9000
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            10.85.41.219         tcp dpt:9000
    3   180 DNAT       tcp  --  *      *       0.0.0.0/0            10.85.41.219         tcp dpt:9000 to:192.168.230.3:8080

Chain CNI-db249f3a7e41a3fb137d5d42 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.228.0/24     /* name: "network_sub" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4          /* name: "network_sub" id: "e06986b4039694a0a10a4b4c62b782ecb83765e92d4a09c2d7f18b651afcf8ee" */

Chain CNI-DN-db249f3a7e41a3fb137d5 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       192.168.228.0/24     10.85.41.219         tcp dpt:9000
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            10.85.41.219         tcp dpt:9000
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            10.85.41.219         tcp dpt:9000 to:192.168.228.3:8080

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Anything further on this?

[mheon]

@Luap99 was looking into a very similar issue on Bugzilla, and preliminary thinking was that it was potentially a kernel issue - reproduced on RHEL 8 only.

[Luap99]

This seems to be a RHEL only issue closing this in favour of https://bugzilla.redhat.com/show_bug.cgi?id=2076310