Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firewalld issue #5568

Closed
JuSt19 opened this issue Mar 20, 2020 · 23 comments
Closed

Firewalld issue #5568

JuSt19 opened this issue Mar 20, 2020 · 23 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue

Comments

@JuSt19
Copy link

JuSt19 commented Mar 20, 2020

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Once we deploy the microservices within podman with selinux and firewalld (both activated)...services start not responding requests properly. If you connect to the container and try to ping, dig any internal or external domain, any action to verify the connection, the pod is not able no do nothing, if we restart the pod but first with firewalld and selinux disabled...everything is ok and all back to normality.

Steps to reproduce the issue:

  1. Activate Firewalld

  2. Deploy two containers:
    podman run --name nginx01 -d -p 8080:80 nginx
    podman run --name nginx02 -d -p 8081:80 nginx

  3. Stop firewalld

  4. Try to make internal DNS query or external such as google or something...

Describe the results you received:

root@4d01a46cd8c7:/# ping 10.88.0.3
PING 10.88.0.3 (10.88.0.3) 56(84) bytes of data.
From 10.88.0.2 icmp_seq=10 Destination Host Unreachable
From 10.88.0.2 icmp_seq=11 Destination Host Unreachable
From 10.88.0.2 icmp_seq=12 Destination Host Unreachable
From 10.88.0.2 icmp_seq=13 Destination Host Unreachable

root@8d9d9097cc6e:/app# ping subdomain.domain.com
ping: subdomain.domain.com: Temporary failure in name resolution

When the microservice try to connec to SQL Server:
SqlExceptionHelper","message":"The connection attempt failed."

Describe the results you expected:

  • The possibility to make dns querys and connect to other services such as sql

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 1.4.4

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.3
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.el7.centos.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 0.3.0, commit: unknown'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 553943040
  MemTotal: 8201318400
  OCIRuntime:
    package: runc-1.0.0-65.rc8.el7.centos.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 5519458304
  SwapTotal: 6509555712
  arch: amd64
  cpus: 4
  hostname: test-podman-01
  kernel: 3.10.0-1062.1.1.el7.x86_64
  os: linux
  rootless: false
  uptime: 594h 16m 31.05s (Approximately 24.75 days)
registries:
  blocked: null
  insecure:
  - 172.16.x.x:8087
  - 172.16.x.x:8089
  search:
  - registry.internaldomain.com
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 16
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 16
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.4.4-4.el7.centos.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 20, 2020
@vrothberg
Copy link
Member

@mheon PTAL

@rhatdan
Copy link
Member

rhatdan commented Mar 20, 2020

I don't think this is an SELinux issue, but I am willing to be proven wrong? If you put the machine in permissive mode, does it still fail with the file wall rules up?

Are there any AVC messages in the audit.log.

This issue looks a lot like a firewall rules.

From the podman info this looks like it is happening with root.

BTW There should be a newer version of podman available for centos.

@ikke-t
Copy link

ikke-t commented Mar 20, 2020

check out it's not the same I had a while back, confusion between iptables and firewalld: #5335

@JuSt19
Copy link
Author

JuSt19 commented Mar 21, 2020

I have updated the main issue with more information and also the title, this issues is focused in firewalld.

I've talked yesterday with @mheon and he asked me to do some tests and here's the output:

[root@test-podman-01 ~]# podman exec -it nginx02 bash
Error: cannot exec into container that is not running: container state improper
[root@test-podman-01 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 256 packets, 18390 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 16 packets, 920 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 153 packets, 17604 bytes)
 pkts bytes target     prot opt in     out     source               destination
[root@test-podman-01 ~]# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 20 packets, 1262 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 12 packets, 802 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 9 packets, 2232 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 17 packets, 2692 bytes)
 pkts bytes target     prot opt in     out     source               destination

When i try to restart the pod and get back to live again, i receive the following error:

[root@test-podman-01 ~]# podman restart nginx02
ERRO[0000] Error deleting network: running [/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.3 -j CNI-7ae4d283fb426dd13357d0c7 -m comment --comment name: "podman" id: "f2d1841e0da4bbea5c94fa8630ed882c4d105ab8ade096901ea6fa00c132a1d7" --wait]: exit status 2: iptables v1.4.21: Couldn't load target `CNI-7ae4d283fb426dd13357d0c7':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
ERRO[0000] Error while removing pod from CNI network "podman": running [/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.3 -j CNI-7ae4d283fb426dd13357d0c7 -m comment --comment name: "podman" id: "f2d1841e0da4bbea5c94fa8630ed882c4d105ab8ade096901ea6fa00c132a1d7" --wait]: exit status 2: iptables v1.4.21: Couldn't load target `CNI-7ae4d283fb426dd13357d0c7':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
ERRO[0000] unable to cleanup network for container f2d1841e0da4bbea5c94fa8630ed882c4d105ab8ade096901ea6fa00c132a1d7: "error tearing down CNI namespace configuration for container f2d1841e0da4bbea5c94fa8630ed882c4d105ab8ade096901ea6fa00c132a1d7: running [/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.3 -j CNI-7ae4d283fb426dd13357d0c7 -m comment --comment name: \"podman\" id: \"f2d1841e0da4bbea5c94fa8630ed882c4d105ab8ade096901ea6fa00c132a1d7\" --wait]: exit status 2: iptables v1.4.21: Couldn't load target `CNI-7ae4d283fb426dd13357d0c7':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n"
Error: cannot listen on the TCP port: listen tcp4 :8081: bind: address already in use

The output after start firewalld again:

 iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  115  6500 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    4   227 INPUT_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    4   227 INPUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    4   227 INPUT_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    3   123 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
   48  3080 FORWARD_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FORWARD_IN_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FORWARD_IN_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FORWARD_OUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FORWARD_OUT_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
   48  3080 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 129 packets, 17341 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
  129 17341 OUTPUT_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD_IN_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 FWDI_public  all  --  enp0s3 *       0.0.0.0/0            0.0.0.0/0           [goto]
   48  3080 FWDI_public  all  --  +      *       0.0.0.0/0            0.0.0.0/0           [goto]

Chain FORWARD_IN_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD_OUT_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination
   48  3080 FWDO_public  all  --  *      enp0s3  0.0.0.0/0            0.0.0.0/0           [goto]
    0     0 FWDO_public  all  --  *      +       0.0.0.0/0            0.0.0.0/0           [goto]

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FWDI_public (2 references)
 pkts bytes target     prot opt in     out     source               destination
   48  3080 FWDI_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FWDI_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FWDI_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FWDI_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FWDI_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FWDI_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FWDO_public (2 references)
 pkts bytes target     prot opt in     out     source               destination
   48  3080 FWDO_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FWDO_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   48  3080 FWDO_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FWDO_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FWDO_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain FWDO_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination
    4   227 IN_public  all  --  enp0s3 *       0.0.0.0/0            0.0.0.0/0           [goto]
    0     0 IN_public  all  --  +      *       0.0.0.0/0            0.0.0.0/0           [goto]

Chain INPUT_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain IN_public (2 references)
 pkts bytes target     prot opt in     out     source               destination
    4   227 IN_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    4   227 IN_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    4   227 IN_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0

Chain IN_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination
    1   104 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW,UNTRACKED

Chain IN_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain IN_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination
iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 60 packets, 3746 bytes)
 pkts bytes target     prot opt in     out     source               destination
   60  3746 PREROUTING_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   60  3746 PREROUTING_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   60  3746 PREROUTING_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain INPUT (policy ACCEPT 1 packets, 104 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 8 packets, 608 bytes)
 pkts bytes target     prot opt in     out     source               destination
    8   608 OUTPUT_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain POSTROUTING (policy ACCEPT 8 packets, 608 bytes)
 pkts bytes target     prot opt in     out     source               destination
    8   608 POSTROUTING_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    8   608 POSTROUTING_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    8   608 POSTROUTING_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination
    8   608 POST_public  all  --  *      enp0s3  0.0.0.0/0            0.0.0.0/0           [goto]
    0     0 POST_public  all  --  *      +       0.0.0.0/0            0.0.0.0/0           [goto]

Chain POSTROUTING_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain POST_public (2 references)
 pkts bytes target     prot opt in     out     source               destination
    8   608 POST_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    8   608 POST_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    8   608 POST_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain POST_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain POST_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain POST_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain PREROUTING_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination
   12   666 PRE_public  all  --  enp0s3 *       0.0.0.0/0            0.0.0.0/0           [goto]
   48  3080 PRE_public  all  --  +      *       0.0.0.0/0            0.0.0.0/0           [goto]

Chain PREROUTING_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain PREROUTING_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain PRE_public (2 references)
 pkts bytes target     prot opt in     out     source               destination
   60  3746 PRE_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   60  3746 PRE_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   60  3746 PRE_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain PRE_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain PRE_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain PRE_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination

@JuSt19 JuSt19 changed the title Firewalld and SELinux Firewalld issue Mar 21, 2020
@JuSt19
Copy link
Author

JuSt19 commented Mar 21, 2020

I have found this links about the same issue in the same env:

#348
#1431 (comment)

@mheon
Copy link
Member

mheon commented Mar 23, 2020

I feel like we may have fixed this on master, but I'm not precisely sure as to when...

@JuSt19 There's a Centos virt-sig repo that distributes more up-to-date Podman packages; would you be willing to give those a shot and see if a more recent version is fixed? @lsm5 where did that repo live, again?

@JuSt19
Copy link
Author

JuSt19 commented Mar 23, 2020

@mheon that's perfect for me, could you please share to me this repo and then i'll test if it's already fixed.

Awaiting!

@lsm5
Copy link
Member

lsm5 commented Mar 23, 2020

I feel like we may have fixed this on master, but I'm not precisely sure as to when...

@JuSt19 There's a Centos virt-sig repo that distributes more up-to-date Podman packages; would you be willing to give those a shot and see if a more recent version is fixed? @lsm5 where did that repo live, again?

I don't maintain the virt-sig repo anymore. All non-fedora and non-rhel is on OBS now. 1.8.2 should be available for centos 7 via the OBS stable repo. See the Kubic project section under CentOS at https://podman.io/getting-started/installation for installation instructions

@JuSt19
Copy link
Author

JuSt19 commented Mar 23, 2020

I feel like we may have fixed this on master, but I'm not precisely sure as to when...
@JuSt19 There's a Centos virt-sig repo that distributes more up-to-date Podman packages; would you be willing to give those a shot and see if a more recent version is fixed? @lsm5 where did that repo live, again?

I don't maintain the virt-sig repo anymore. All non-fedora and non-rhel is on OBS now. 1.8.2 should be available for centos 7 via the OBS stable repo. See the Kubic project section under CentOS at https://podman.io/getting-started/installation for installation instructions

Still failing after update to 1.8.2 and doing the same thing to reproduce the same with the old version.

@cfelder
Copy link
Contributor

cfelder commented Apr 9, 2020

Hello, is there any progress on this? I am experiencing the same issue on CentOS 8. When firewalld is enabled. Basically name resolution does not work whereas routing works fine when using ips directly.

@mheon
Copy link
Member

mheon commented Apr 9, 2020

@mccv1r0 Any thoughts on this one?

@cfelder
Copy link
Contributor

cfelder commented Apr 9, 2020

This may be related to #5335. I am currently not using the kubic repos but with the version provided in AppStream repos for CentOS 8

podman-1.6.4-2.module_el8.1.0+272+3e64ee36.src.rpm

a wrong CNI configuration is created containing iptables as backend for the firewall plugin.

This is easy to reproduce using:

$ podman network create newnetwork
$ podman network inspect newnetwork | grep backend
				"backend": "iptables",

when removing the backend line name resolution works fine.

@baude
Copy link
Member

baude commented Apr 9, 2020

this is a known issue and is fixed upstream ... both the network create code and default cni config

@cfelder
Copy link
Contributor

cfelder commented Apr 9, 2020

any chance getting this into the releases shipped with the base/appstream repos on rhel?

@rhatdan
Copy link
Member

rhatdan commented Apr 9, 2020

Is it fixed in podman-1.16.4? Which should be released in rhel8.2 release.

@rhatdan
Copy link
Member

rhatdan commented Apr 9, 2020

We will release podman-1.9.* in RHEL8.2.1 in the summer time.

@cfelder
Copy link
Contributor

cfelder commented Apr 9, 2020

Looking forward to see this and hopefully dnsmasq plugin in the next release. Meanwhile I can workaround this by myself.

Thanks a lot.

@mheon
Copy link
Member

mheon commented Apr 9, 2020

Can you confirm whether there is a firewall block in the base network we ship (87-podman-bridge.conflist)?

@cfelder
Copy link
Contributor

cfelder commented Apr 9, 2020

For podman-1.6.4-2.module_el8.1.0+272+3e64ee36.src.rpm there is the following block:

        {
            "type": "firewall",
            "backend": "iptables"
        }

@mheon
Copy link
Member

mheon commented Apr 9, 2020

Alright. So we are at least shipping a vaguely working config, here.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jun 9, 2020

@mheon Is this still an issue with the upstream?

@mheon
Copy link
Member

mheon commented Jun 9, 2020

This should be fixed on upstream CNI configs.

@mheon mheon closed this as completed Jun 9, 2020
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue
Projects
None yet
Development

No branches or pull requests

9 participants