/etc/resolv.conf nameserver order varies when starting a container with multiple networks.

[ykuksenko]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When starting a container with two networks attached the nameserver order inside the container /etc/resolv.conf file seems to be random. Unfortunately this causes some name resolution issues. I was expecting the order to be the same as what the --network argument received.

Specifically I have traefik -> nextcloud -> postgres

• traefik has its own network (traefik)
• nextcloud uses traefik and nextcloud networks
• postgres uses just the nextcloud network
  Nextcloud fails to resolve nextcloud-postgres.dns.podman when the traefik name server is listed first in /etc/resolv.conf

Simpler steps are provided to reproduce the issue.

Steps to reproduce the issue:

watch -d -n 1 podman run --rm -ti --network nextcloud,traefik fedora cat /etc/resolv.conf

Describe the results you received:
I see name servers for the two container networks switch places while my host nameserver always comes in last.

Describe the results you expected:
I expected the nextcloud network nameserver to be listed first, followed by the traefik network nameserver followed by the host nameserver.

Additional information you deem important (e.g. issue happens only occasionally):

• This happens randomly as mentioned originally.
• Running containers as root - not sure if this makes a difference.
• both networks use the bridge driver (in podman 3.x I had the dns plugin enabled/listed, that does not seem to be listed in podman 4.x)

Output of podman version:

# podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18
Built:        Fri May  6 16:15:54 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

# podman info --debug
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 96.39
    systemPercent: 1.27
    userPercent: 2.35
  cpus: 2
  distribution:
    distribution: fedora
    variant: cloud
    version: "36"
  eventLogger: journald
  hostname: container
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.6-300.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 323362816
  memTotal: 12177186816
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8129605632
  swapTotal: 8589930496
  uptime: 13h 17m 25.05s (Approximately 0.54 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 28
    paused: 0
    running: 26
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 42313187328
  graphRootUsed: 35236487168
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 136
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1651853754
  BuiltTime: Fri May  6 16:15:54 2022
  GitCommit: ""
  GoVersion: go1.18
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.1.0-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
running in a libvirt vm.

Edit: fixed typo: postgres container uses the nextcloud network not traefik

[mheon]

Using the netavark backend will alleviate this (server order is still random, but the servers are identical and can respond for all networks).

[ykuksenko]

I will see about trying that out. It seems like I need to recreate all containers and networks for that. So it will take a just a bit.
Is the random nameserver order expected (ie not a bug) or is this something that will probably get changed (fixed)?

[Luap99]

The random order is by design since the info is stored in a map. In theory we could make it deterministic by sorting the nameservers before adding them.

[mheon]

Even then, I don't know if it's worth ensuring sorting during all the operations that effect resolv.con (I'm thinking of network connect/disconnect - do we sort again when adding?

[Luap99]

I don't know. The current idea is to always add the new server in front on network connect.

[ykuksenko]

I rebuilt my test infrastructure to use netavark. (Full OS reinstall and replaced all containers)
I think every thing is using the netavark backend at this point. Let me know if you need any additional output to verify that.

# podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18
Built:        Fri May  6 16:15:54 2022
OS/Arch:      linux/amd64

# rpm -q netavark podman
netavark-1.0.3-3.fc36.x86_64
podman-4.1.0-1.fc36.x86_64

# podman info  --debug
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 97.24
    systemPercent: 0.93
    userPercent: 1.83
  cpus: 2
  distribution:
    distribution: fedora
    variant: cloud
    version: "36"
  eventLogger: journald
  hostname: container.redacted
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.5-300.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 148381696
  memTotal: 6217089024
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 5851836416
  swapTotal: 6217003008
  uptime: 15h 16m 15.62s (Approximately 0.62 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 19
    paused: 0
    running: 19
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 41788899328
  graphRootUsed: 9318744064
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 67
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1651853754
  BuiltTime: Fri May  6 16:15:54 2022
  GitCommit: ""
  GoVersion: go1.18
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

I reran my dns resolution tests and still had the same results. This time I found some additional info. Since I rebuilt my test infrastructure I do not know if this issue happens with the CNI backend also.

I looked deeper into the issue and found that with netavark I get fqdn dns reponses from only one network but not both.

Setup for the tests:

podman network create --subnet 192.168.55.0/24 inner
podman network create --subnet 192.168.56.0/24 outer
podman run --detach --rm -ti --name left --network inner alpine sleep 9000

FQDN test

podman run --rm -ti --name right --network inner,outer alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<dig left network'; dig left.dns.podman +short @192.168.55.1; echo '>>>dig right network'; dig left.dns.podman +short @192.168.56.1"

FQDN result

search dns.podman dns.podman
nameserver 192.168.55.1
nameserver 192.168.56.1
<<<dig left network
192.168.55.28
>>>dig right network

You will notice that the right network has no response.

SHORTNAME test

podman run --rm -ti --name right --network inner,outer alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<dig left network'; dig left +short @192.168.55.1; echo '>>>dig right network'; dig left +short @192.168.56.1"

SHORTNAME result

search dns.podman dns.podman
nameserver 192.168.56.1
nameserver 192.168.55.1
<<<dig left network
192.168.55.28
>>>dig right network
192.168.55.28

Here both networks responded was described above. (Ie same result from all networks)

Other notes:

• I removed my local DNS suffixes and name server from all outputs.
• Repeating these test yields the same results.
• All of these are run from the root account

Questions:

• Is the FQDN behavior expected?
• Should I now file a new issue for this?

[mheon]

FQDN behavior definitely seems like a bug, please file a fresh issue for it.

[ykuksenko]

In researching containers/aardvark-dns#403 I found that docker only provides one name server for multiple networks to all containers so server order is not a problem there. Though docker seems to not allow creating a container with multiple networks from the start.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon @Luap99 What should we do with this one?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon @Luap99 whats up with this?

[Luap99]

We could sort the nameservers by name to make it deterministic.

[rhatdan]

SGTM

[felixsanz]

aren't this and containers/aardvark-dns#403 the same bug? should this be closed in favor of it?

In fact this issue was first reported early 2021 here: #9492. So this one is the same too (but this is closed)

[ykuksenko]

I do not see these bugs as being the same. At the moment this bug just makes the other bug randomly visible. I believe this bug would still affect anyone using CNI plugins, even if the other was resolved. This is presuming that ultimately podman, not netavark or CNI plugins generate the contents of /etc/resolv.conf.

If CNI plugins are dropped, as planned, and this aspect is not addressed you would still be generating a randomly ordered /etc/resolv.conf config. This will likely lead to questions and confusion when troubleshooting other network issues in the future.

[Luap99]

I am closing this as wontfix. While with with netavark the order is still not deterministic it should behave correctly as aardvark-dns can resolve names for all networks unlike the old dnsname plugin.
Given CNI is deprecated will will never fix that there.
And for deterministic ordering of the networks we already have: #12850