Port forwarding not working when docker is active

[bverhagen]

Hi,

I am not sure this is the right place to ask nor do I know whether this is unexpected for you, but I would like to get your point of view on the following problem I encountered:

I am in the middle of moving Docker container workloads to podman 4.3.1 using netavark as network backend. Some containers are still running in docker, while another one, a postgresql database that for dubious reasons exposes its database to the host port and to another container on its own network, has been moved to podman. When Docker is not running, assigning the database container to the network, while publishing port 5432 on the host using -p 5432:5432 works exactly as expected (meaning: pointing an external postgresql client to the host works as if the database is running on the host). However, when I start Docker, the port forward on the host to the podman container stops working. Connecting directly to the database container still works. None of the containers still running in Docker publish to the host's port (not to port 5432 nor to any other one).

My guess is that Docker overwrites the port forward in the firewall to the podman container. I do not know how to debug this further or fix this though. Do you have ideas/remarks/workarounds that could support this hybrid situation until all of them have been moved to podman?

[Luap99]

Can you share the output of iptables -nvL and iptables -nvL - nat when this happens? I guess there are some conflicting rules in here.

[bverhagen]

Thank you for following up on this! After a couple more restarts and changes, I do not seem to be able to reproduce it any longer.

Closing the issue, my apologies to have disturbed you with this.

[bverhagen]

It looks like the issue is back now:

# iptables -nvL
Chain INPUT (policy ACCEPT 542 packets, 39542 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination         
2035K  222M NETAVARK_FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* netavark firewall plugin rules */
1517K  136M DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
1517K  136M DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 723K   57M ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
 723K   69M ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
51548 8889K ACCEPT     all  --  br-af3d99ee7d86 br-af3d99ee7d86  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 521 packets, 48028 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 723K   69M DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      br-af3d99ee7d86 !172.18.0.0/16        0.0.0.0/0           
19897 1174K DROP       all  --  br-af3d99ee7d86 *       0.0.0.0/0           !172.18.0.0/16       
1497K  135M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
 723K   69M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
1517K  136M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain NETAVARK_FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 269K   61M ACCEPT     all  --  *      *       0.0.0.0/0            10.89.0.0/24         ctstate RELATED,ESTABLISHED
 249K   25M ACCEPT     all  --  *      *       10.89.0.0/24         0.0.0.0/0           

Chain vl (0 references)
 pkts bytes target     prot opt in     out     source               destination

The nat option does not seem to work:

# iptables -nvL - nat
Bad argument `-'
Try `iptables -h' or 'iptables --help' for more information.

[Luap99]

sorry the correct command is iptables -nvL -t nat

[xingd]

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2043 87676 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
 1993 85736 NETAVARK-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
    3   180 NETAVARK-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1100 70444 NETAVARK-HOSTPORT-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    2   123 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    0     0 NETAVARK-1D8721804F16F  all  --  *      *       10.88.0.0/16         0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           

Chain NETAVARK-1D8721804F16F (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            10.88.0.0/16        
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4         

Chain NETAVARK-DN-1D8721804F16F (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 NETAVARK-HOSTPORT-SETMARK  tcp  --  *      *       10.88.0.0/16         0.0.0.0/0            tcp dpt:6984
    0     0 NETAVARK-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:6984
  131  7900 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:6984 to:10.88.0.4:5984

Chain NETAVARK-HOSTPORT-DNAT (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  131  7900 NETAVARK-DN-1D8721804F16F  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:6984 /* dnat name: podman id: e27d2262233abc31a6541b83ef841615580caee66ea6ac5d5e0f1be8ee9dc1d0 */

Chain NETAVARK-HOSTPORT-MASQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* netavark portfw masq mark */ mark match 0x2000/0x2000

Chain NETAVARK-HOSTPORT-SETMARK (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x2000

I try to use podman and docker side by side, progressively migrate to podman.

[baude]

can we close this or shall we leave it open? im not seeing any conclusion here.

[bverhagen]

I switched completely to podman in the meantime, which resolves the issue for me.

[sumpfralle]

The issue seems to be caused by the firewall configuration applied by Docker.
Specifically Docker changes the policy of the FORWARD chain to DROP.

But sadly netavark relies on the policy to be ACCEPT, since it does not add any specific rule, which allows DNATed packets to be actually FORWARDed.
This seems to be the cause of friction for the author of this issue (and for me).

The following rule fixes it for me:

iptables -A NETAVARK_FORWARD -m conntrack --ctstate DNAT -j ACCEPT

The other two FORWARD rules in NETAVARK_FORWARD only deal with outgoing traffic.
But the incoming traffic (via port forwardings) is not handled at the moment. Thus, the default FORWARD policy needs to handle it.

Removing Docker from the system fixes the issue, since the policy of the FORWARD chain remains at its default state of ACCEPT.
But it is certainly very problematic for netavark to rely on such an "open" configuration.

Thanks for your time!

[sumpfralle]

@bverhagen: I guess, this issue (the missing forward rule) is still open?

[bverhagen]

@sumpfralle : thanks for your consideration and for providing the real solution! As mentioned, I pushed through and converted everything to podman right away.

I hope it will help others though that do not have this luxury!