Regression 3.4: missing entries in /etc/hosts

[douglas-legulas]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Example: a pod starts 3 containers named example-mariadb, example-php and example-httpd. Because their /etc/hosts file is missing the required entries, they cannot locate one another through hostname resolution. In this example, PHP fails to locate the MariaDB container by hostname example-mariadb. As a symptom PHP logs the error:

php_network_getaddresses: getaddrinfo failed: Name or service not known

/etc/hosts available inside containers before v3.4:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.88.0.4 example 081aafb26c65-infra
127.0.1.1 example example-mariadb
127.0.1.1 example example-httpd
127.0.1.1 example example-php

/etc/hosts available inside containers in v3.4:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.88.0.5 example 081aafb26c65-infra
10.88.0.1 host.containers.internal

Changes in v3.4 that may have caused the regression: #11411, #11596

Steps to reproduce the issue:

1. Build and start the pod at https://1drv.ms/u/s!ApqEOOKoIKfoeo7T9Yp5pk5cM9U?e=c0Q086

2. Exec into the shell of one of the containers and check its /etc/hosts.

Output of podman version:

Version:      3.4.0
API Version:  3.4.0
Go Version:   go1.16.8
Built:        Thu Sep 30 16:40:21 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: workstation
    version: "34"
  eventLogger: journald
  hostname: alq22
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.11-200.fc34.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 9558335488
  memTotal: 16698474496
  ociRuntime:
    name: crun
    package: crun-1.2-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.2
      commit: 4f6c8e0583c679bfee6a899c05ac6b916022561b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 59m 50.09s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 4
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 11
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.0
  Built: 1633030821
  BuiltTime: Thu Sep 30 16:40:21 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.0-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora Workstation 34 (amd64).

[zhangguanzhang]

I cannot access the url: https://1drv.ms/u/s!ApqEOOKoIKfoeo7T9Yp5pk5cM9U?e=c0Q086
could you paste the content to here？

[o-alquimista]

Sure.

LegulasPod.zip

[WoollyMammal]

I was just about to post my own report on pod-internal containers failing to resolve hostnames, but while filling out the bug report I learned it was an issue with the hosts file. Rechecking the issues for "hosts" and... this issue already exists. sigh

If you need more details, or just want another sample, let me know and I'll post my details here too.

[douglas-legulas]

@zhangguanzhang I forgot to mention that @o-alquimista is another account of mine. Sorry about that.

[rhatdan]

Could you show the podman command used to create the /etc/hosts file?

[douglas-legulas]

Could you show the podman command used to create the /etc/hosts file?

Sorry, what do you mean? The pod building scripts are here: https://github.com/containers/podman/files/7388549/LegulasPod.zip

[rhatdan]

Could you make a simple repeater that we could examine.

[douglas-legulas]

I've added a virtual host and an example webpage.
LegulasPod.zip

To reproduce:

1. Build the pod (as root)

$ cd LegulasPod
# sh ./scripts/build.sh

2. Check /etc/hosts:

# podman exec legulas-php cat /etc/hosts

Expected results (before v3.4):

legulas c7d7ce974eb3-infra
10.88.0.1 host.containers.internal
127.0.0.1 legulas legulas-httpd
127.0.0.1 legulas legulas-php
127.0.0.1 legulas legulas-mariadb

Actual results (now with v3.4):

legulas c7d7ce974eb3-infra
10.88.0.1 host.containers.internal

[rhatdan]

@Luap99 PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.