[Bug]: host's /etc/hosts file overrides private network dns settings in podman

[hcldan]

Issue Description

Moving over to a new machine with podman instead of docker, still ironing out the issues using docker-compose with podman.

I used to have a custom domain let's call it foo.bar.com, car.bar.com, etc mapped to 127.0.0.1 in my host's /etc/hosts file.
In the docker-compose.yml I had a private network with those same hosts set.
Inside the containers attached to the network, they would get the private network address, using podman they get 127.0.0.1

Steps to reproduce the issue

on host machine:

127.0.0.1 foo.bar.com proxy.bar.com

docker-compose.yml:

version: '3.3'

services:
  proxy:
    image: nginx:stable-alpine
    ports:
      - 80:80
      - 443:443
    hostname:  proxy.bar.com
    networks:
      mynet:
        aliases:
          -  proxy.bar.com
  foo:
    image: some-server:latest
    ports:
      - 9555:9555
    hostname:  foo.bar.com
    networks:
      mynet:
        aliases:
          -  foo.bar.com
    volumes:
networks:
  mynet:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 10.10.0.0/16

Describe the results you received

exec into proxy.bar.com

ping foo.bar.com

get 127.0.0.1

Describe the results you expected

I expected an ip inside the mynet address space

podman info output

host:
  arch: amd64
  buildahVersion: 1.27.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: 56561007b6a59ea175ee9a67384639721499e160'
  cpuUtilization:
    idlePercent: 99.42
    systemPercent: 0.23
    userPercent: 0.36
  cpus: 20
  distribution:
    distribution: '"rhel"'
    version: "9.1"
  eventLogger: journald
  hostname: frisbee
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-162.6.1.el9_1.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 26817802240
  memTotal: 66804899840
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.5-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.el9_0.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 33592176640
  swapTotal: 33592176640
  uptime: 41h 58m 30.00s (Approximately 1.71 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/ddumont/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 1
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ddumont/.local/share/containers/storage
  graphRootAllocated: 988412391424
  graphRootUsed: 45282320384
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  volumePath: /home/ddumont/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1666809014
  BuiltTime: Wed Oct 26 14:30:14 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

RHEL 9.1

Additional information

No response

[Luap99]

Can you provide podman info please as requested in the issue template.
Note: If you use RHEL it is recommend to go trough the Red Hat support channels and not github, e.g. create a bugzilla.

By default podman will copy all host entries from the host into the container. so this is expected. If you want to change that behaviour you can use the base_hosts_file option in containers.conf, see https://github.com/containers/common/blob/main/docs/containers.conf.5.md

[hcldan]

@Luap99 Added podman info output, sorry I missed that.
Also, I do have a bugzilla account so I'll go though there from now on.

I was reading a bunch of articles on how podman tries to support docker and docker-compose apis as best it can... So I figured I would file this as the behavior you describe doesn't happen in docker for me... So maybe there should be a different default when run from docker-compose? Not sure, but thank you for the info on the containers.conf!

[Luap99]

Yeah you are right we could choose a better default for the docker API.
@mheon WDYT?

for reference I think we lost tracked of this in #13748

[mheon]

Sure, matching Docker seems easy enough here given we already added support for various ways to use /etc/hosts

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon @Luap99 Reminder

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc @Luap99 PTAL

[Luap99]

@rhatdan This is a nice to have, I have much more higher priority stuff to do.

It would be great if someone from the community could work on it.
This is needed for it to work:

1. Add the base_hosts_file option to the container spec and config so it can be set for specific containers.
2. Then make sure the docker compat api sets this field to none to match the docker behavior.

[vincentywdeng]

Hi @Luap99 and @hcldan , I found this is fixed in main branch now by adding CONTAINERS_CONF environment variable in systemd unit file

/usr/local/lib/systemd/system/podman.service:

[Service]
Delegate=true
Type=exec
KillMode=process
Environment=LOGGING="--log-level=info" CONTAINERS_CONF="/home/ywdeng/.config/containers/containers.conf"
ExecStart=/home/ywdeng/Desktop/go/src/github.com/containers/podman/bin/podman $LOGGING system service

/home/ywdeng/.config/containers/containers.conf:

[containers]
base_hosts_file="/home/ywdeng/Desktop/hosts"

home/ywdeng/Desktop/hosts:

127.0.0.1  vincent.test

Test:

sudo docker-compose exec foo cat /etc/hosts
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
127.0.0.1	vincent.test
127.0.0.1	localhost
::1	localhost
10.10.0.1	host.containers.internal host.docker.internal
10.10.0.6	foo.bar.com desktop_foo_1

So maybe simply adding a document about how to setting it?

[gavinkflam]

I worked on a fix by applying @Luap99's suggestions. Please help reviewing the PR #21013.

@vincentywdeng CONTAINERS_CONF, containers.conf, and base_hosts_file are already documented here.