DNS not working with containers on a single network.

[icsy7867]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. Create a podman network
   podman network create --name external

2. Create two pods on the same network (Mine are nginx and a mysql container). Also install iputils-ping in them. Lets call them some-nginx and some-mysql

3. try to ping one another using their container names:
   podman exec -it some-nginx bash
   ping some-mysql or ping some-mysql.dns.podman

Describe the results you received:
Temporary failure in name resolution

Describe the results you expected:
Successful pings

Additional information you deem important (e.g. issue happens only occasionally):
I have also tried to add --hostname to the deployments. Same results.

Output of podman version:

Version:      3.0.2-dev
API Version:  3.0.0
Go Version:   go1.15.13
Built:        Tue Jun  8 03:52:06 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.8
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-3.module+el8.4.0+11311+9da8acfb.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: a35bb9ea67d5a83c7da53202f2fcd505c036d29c'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: bodewadmi.icl.gtri.org
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-305.10.2.el8_4.x86_64
  linkmode: dynamic
  memFree: 519438336
  memTotal: 4119433216
  ociRuntime:
    name: runc
    package: runc-1.0.0-73.rc93.module+el8.4.0+11311+9da8acfb.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.13
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2113671168
  swapTotal: 2147479552
  uptime: 21h 21m 39.94s (Approximately 0.88 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 25
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1623138726
  BuiltTime: Tue Jun  8 03:52:06 2021
  GitCommit: ""
  GoVersion: go1.15.13
  OsArch: linux/amd64
  Version: 3.0.2-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.1-7.module+el8.4.0+11311+9da8acfb.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

[icsy7867]

Doing some poking around, and it looks like our RHEL is pushing 3.0.2-Dev which is an odd thing to push by default so I am looking into that.

I have also noticed that the DNS plugin is not present.

sudo podman network create test
sudo podman network inspect test

Show that there is no DNS plugin. So I tried adding it into the conflist file:
sudo nano /etc/cni/net.d/test.conflist

After doing so sudo podman network inspect test seems happy, but when I try to launch a container using this new network I see:
WARN[0000] Error validating CNI config file /etc/cni/net.d/test.conflist: [failed to find plugin "dnsname" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]]

SO I am looking to see if this is something I can install manually.... Any advise welcomed!

[flouthoc]

@icsy7867 try

git clone https://github.com/containers/dnsname.git
cd dnsname
go build -a
cp dnsname /usr/libexec/cni

[icsy7867]

@icsy7867 try

git clone https://github.com/containers/dnsname.git
cd dnsname
go build -a
cp dnsname /usr/libexec/cni

Thank you! I'll try this.

=======
Update

I tried this... I received an error:
no Go files in /build/dnsname

Also, I tried to follow the directions here:
https://github.com/containers/dnsname/blob/main/README_PODMAN.md
(Which seems to also install using GO) and I receive errors:

install -Z -d -m 755 /usr/libexec/cni
install -Z -m 755 bin/dnsname /usr/libexec/cni/dnsname
install: cannot stat 'bin/dnsname': No such file or directory
make: *** [Makefile:76: install] Error 1

=====
Update 2

Looks like I just needed to change into:
/dnsname/plugins/meta/dnsname/dnsname

go build -a worked after that..

=========
Update 3

New problem. Things seemed OK. I created a new podman network (Which seemed to add all the correct dnsname bits), and then I tried launch a podman container using the new network and was greated with:

sudo podman network create test
sudo podman run -it --network=test -d nginx
WARN[0000] Error validating CNI config file /etc/cni/net.d/test.conflist: [netplugin failed: "/usr/libexec/cni/dnsname: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /usr/libexec/cni/dnsname)\n"]
ERRO[0000] CNI network "test" not found
Error: error configuring network namespace for container de8c83ffc5455913846ea18cbe325e357d1dfdaccd82a43776f3ede13de4b954: CNI network "test" not found

It looks like I need a newer version of glibc. Looks like our RHEL Satellite is pushing out 2.28.

[flouthoc]

yup you can update glibc as well.

[Luap99]

The dnsname plugin can be installed with dnf install podman-plugins. There is no need to build from source.

[flouthoc]

yes you can install from dnf as well, but i think you'll still need to update glibc if that is not updated as a dependency.

[patricmutwiri]

For debian users, check this out: https://packages.debian.org/sid/golang-github-containernetworking-plugin-dnsname