host.containers.internal is inconsistent for rootful containers assigned to a pod and not

[lovette]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When rootless, the IP assigned to host.containers.internal is consistent between pod and non-pod containers and refers to the host "public IP".

When rootful, host.containers.internal entries are not consistent.

• For non-pod containers it refers to the container network gateway (10.x.0.1).
• For containers assigned to a pod, there are two entries: one assigned to the container network gateway (10.x.0.1) and another with the host "public IP" (172.16.x.x in my case.)

This leads me to wonder- Is it a bug that rootful containers have such wildly different host entries? If not, why have the same name assigned to two IPs? And why do non-pod containers not get an entry for the host "public IP"?

Steps to reproduce the issue:

*** Rootless ***

$ podman run -it --rm --name mycontainer ubi8 cat /etc/hosts
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1		localhost localhost.localdomain localhost6 localhost6.localdomain6
# used by slirp4netns
10.0.2.100	d8c2d34abb9c mycontainer
172.16.209.6	host.containers.internal

$ podman pod create --name mypod
$ podman run -it --rm --name mycontainer --pod mypod ubi8 cat /etc/hosts
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1		localhost localhost.localdomain localhost6 localhost6.localdomain6
# used by slirp4netns
10.0.2.100	mypod e8881f2e97fa-infra
172.16.209.6	host.containers.internal

*** Rootful ***

$ sudo podman run -it --rm --name mycontainer ubi8 cat /etc/hosts
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1		localhost localhost.localdomain localhost6 localhost6.localdomain6
10.88.0.20	d843fa3f3fc7 mycontainer
10.88.0.1	host.containers.internal

$ sudo podman pod create --name mypod
$ sudo podman run -it --rm --name mycontainer --pod mypod ubi8 cat /etc/hosts
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1		localhost localhost.localdomain localhost6 localhost6.localdomain6
10.88.0.17	mypod e81a6778d16d-infra
10.88.0.1	host.containers.internal
172.16.209.6	host.containers.internal

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.0.0-rc5
API Version:  4.0.0-rc5
Go Version:   go1.17.5

Built:      Sat Feb 12 01:12:41 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
	package: conmon-2.1.0-1.el9.x86_64
	path: /usr/bin/conmon
	version: 'conmon version 2.1.0, commit: 8ef5de138efb6f0aad657082cdea22cf037792cb'
  cpus: 2
  distribution:
	distribution: '"centos"'
	version: "9"
  eventLogger: journald
  hostname: dd-grogu-dev
  idMappings:
	gidmap: null
	uidmap: null
kernel: 5.14.0-47.el9.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2074648576
  memTotal: 4073611264
  networkBackend: netavark
  ociRuntime:
	name: crun
	package: crun-1.4.2-1.el9.x86_64
	path: /usr/bin/crun
	version: |-
	  crun version 1.4.2
	  commit: f6fbc8f840df1a414f31a60953ae514fa497c748
	  spec: 1.0.0
	  +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
	exists: true
	path: /run/podman/podman.sock
  security:
	apparmorEnabled: false
	capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
	rootless: false
	seccompEnabled: true
	seccompProfilePath: /usr/share/containers/seccomp.json
	selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
	executable: /bin/slirp4netns
	package: slirp4netns-1.1.8-3.el9.x86_64
	version: |-
	  slirp4netns version 1.1.8
	  commit: d361001f495417b880f20329121e3aa431a8f90f
	  libslirp: 4.4.0
	  SLIRP_CONFIG_VERSION_MAX: 3
	  libseccomp: 2.5.2
  swapFree: 2146430976
  swapTotal: 2147479552
  uptime: 50h 19m 42.78s (Approximately 2.08 days)
plugins:
log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
	number: 15
	paused: 0
	running: 15
	stopped: 0
  graphDriverName: overlay
  graphOptions:
	overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
	Backing Filesystem: xfs
	Native Overlay Diff: "false"
	Supports d_type: "true"
	Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
	number: 24
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.0-rc5
  Built: 1644628361
  BuiltTime: Sat Feb 12 01:12:41 2022
  GitCommit: ""
  GoVersion: go1.17.5
  OsArch: linux/amd64
  Version: 4.0.0-rc5

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.0.0-0.7.rc5.el9.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Running on CentOS 9 Stream in a VMware Fusion VM.

[mheon]

I bet this is related to subsequent containers in the pod adding a host.containers.internal entry, despite there already being one from the infra container.

[Luap99]

@mheon We should discuss a complete /etc/hosts handling rewrite. IMO we should move it into c/common so that buildah can use it to. Also it has to support adding/removing entries from an already existing file.

The first step would be to come up with a proper description on what entries should be in there with a given network mode and pod. Right now we always change a tiny thing and it keeps breaking something else.

[mheon]

SGTM - this sounds like a good way to clean things up

[Luap99]

OK I will create a proposal this week.

[lovette]

While we're on the subject of /etc/hosts, it just came to my attention (I'm new to podman...) that the host /etc/hosts is copied into each container. This exposes the container to way more information than I want it to have access to. I'd love to have more control over the content of container /etc/hosts. It's helpful that some default hosts are added automatically, so --no-hosts is too heavyhanded in my case.

Brainstorming here...

• --hosts-no-hostfile to prevent starting with the host /etc/hosts
• --hosts-hostfile=FILE to use a custom hosts FILE as the starting point
• --hosts-append=FILE to append hosts FILE to /etc/hosts
• --hosts-add-host=host:alias to add an alias for a host to /etc/hosts

Options like --network-alias are helpful but I want to limit some host changes to individual containers.

[rhatdan]

Why not just volume mount in the hosts file that you want. Otherwise I see the options getting out of hand.

[lovette]

I had the same thought and agree with the sentiment. It is very helpful to give podman some freedom to manipulate the hosts file to add entries such as host.containers.internal that may only be known at runtime. Currently, using a volume mount has the same effect as --no-hosts, in which case podman makes no host changes. I'm wondering if a middle ground can be found, where podman keeps some freedom to make changes and more control is given to the user. And if the related code is due for a rewrite anyway, I figure now's a good time to consider it 😁

[rhatdan]

How about --no-hosts --add-hosts ....

Which would eliminate the hosts /etc/hosts and would force the user to add his own.

[lovette]

To me --add-hosts implies appending something (see below.) How about just --hosts?

This seems fairly versatile:

Option	/etc/hosts source	Auto entries
Current default	Host	Yes
--hosts=FILE	FILE	Yes
--hosts=""	Image	Yes
--no-hosts	Image	No
--hosts=FILE --no-hosts	FILE	No

Auto entries include manipulations such as adding the container’s own IP address and any hosts from --add-host.

In addition, changing --add-host to take a FILE argument or a new option --add-hosts=FILE would simplify occasions where multiple --add-host arguments are necessary.

Lastly- All my containers are in various pods assigned to a custom network. What I see now is --add-host for one container adds the host for all my containers. Is it possible to limit host additions to specific containers?

[rhatdan]

Containers on the same network share the same /etc/hosts and /etc/resolv.conf file.

[rhatdan]

I modified your table, to match what I would expect.

[rhatdan]

@vrothberg @mheon @Luap99 WDYT?

[lovette]

Containers on the same network share the same /etc/hosts

I know that's the case today, but is this a nonnegotiable tenet of container networking? 😁 The scenario I'm working with is my app connects to services by hostname. I want to run two "identical" containers, each connecting to a different pod running the service endpoint. To achieve this, I need each container to have it's own service name --> pod ip host entry.

[rhatdan]

You want them in the same pod?

[lovette]

I lean towards putting all containers into pods so in this case I'd have four pods, each with one container, all on the same custom network. Two "front end" pods and two "service" pods. In essence, what would be ideal is if --add-host for a container took precedence over any hosts configuration for the pod or network. My first thought is perhaps the --hosts=FILE option would also imply that /etc/hosts is not network-wide. But taking a step back and looking at the option table above, it seems like that would be the assumption anyway. If you have N containers all with different hosts options, and they're all propagated up to the network, what a mess you end up with 😁 As it stands today, it's counterintuitive that --add-host for one container/pod implies adding the host to all containers/pods. This is in contrast to options like --network-alias which is clear that it affects the entire network.

[Luap99]

I am against adding new cli options.
We should disscus this at the next cabal meeting IMO.

[lovette]

Are those meetings at 11 am ET as mentioned on podman.io or 10 am as mentioned in issue #10670?

[Luap99]

@lovette The meeting is this thursday at 11 am ET. https://podman.io/community/meeting/#next-meeting-thursday-march-17-2022-1100-am-edt-utc-4

[lovette]

Since the Community Cabal meeting yesterday I've been trying to reconcile the operational differences between using the default network and custom networks. On the face of it, one would think that moving from one to the other would simply give you more control over network properties. But in fact fundamental changes occur. Initially I was disappointed to learn .NetworkSettings fields are only populated for the default network (#10232). Through the discussion of this issue, I realize that /etc/hosts management is also completely different.

Given this, the table above (proposing new options, but also documenting how /etc/hosts is managed) needs a few more dimensions, such as:

• Is the container on the default network or a custom network?
• What network handler is in place - slirp4netns, bridge, aardvark-dns?
• Does the container image have an /etc/hosts? Is it empty?
• Rootful or rootless?

I contend there shouldn't be so much variance between default and custom networks and as such the effect of --add-host should be consistent regardless of what network a container is on.

As things stand now, the off-the-cuff solution for the scenario I describe above is to put containers that need specific hosts into their own network. But managing N containers on N+1 networks and bridging traffic between them is, if possible at all, a non-trivial amount of work.

As for my proposed options, if adding one like --hosts=FILE is a non-starter, would an alternative be combining a COPY custom_hosts /etc/hosts with multiple --add-host arguments and a --no-hosts that controls whether automatic entries are added?

[Luap99]

There is no difference between containers on default or custom network! There should also be no difference for rootless/rootful. Please provide reproducers if you see actual differences.

You will see differences for --net= host, bridge, slirp4netns or none, this is expected.

[lovette]

Well, I swear I experienced a container --add-host affecting unrelated containers, but I cannot reproduce that today. 🤷‍♂️ After Dan mentioned that containers on the same network share the same /etc/hosts I didn't investigate further. I felt confident in what I saw at the time and can only explain away my confusion by the fact that removing and restarting a container causes duplicate hosts, and I'd stopped and started containers countless times! 🤓

% podman version
Client:		Podman Engine
Version:	4.0.2
API Version:	4.0.2
Go Version:	go1.17.5
Built:		Thu Mar  3 14:56:51 2022
OS/Arch:	linux/amd64

% sudo podman network create --subnet 10.90.0.0/24 foonetwork
foonetwork

% sudo podman pod create --name fooredis-pod --network foonetwork
ec5bca162043123a0fece2d4c7f80002bb26bd9e404efe0a339efc4d1a829bbb

% sudo podman run --detach --name fooredis --pod fooredis-pod --add-host foo:10.90.0.100 redis
aed7a8d14c00c747213a99b04bffe5a2ea504f87ef5e885ee2e249fd3f1ed848

% sudo podman exec -it fooredis cat /etc/hosts
contents of host /etc/hosts
...
10.90.0.6       fooredis-pod ec5bca162043-infra
10.90.0.1 	host.containers.internal
10.90.0.100 	foo
172.16.209.6    host.containers.internal

% sudo podman pod create --name barredis-pod --network foonetwork
0ed9be464360d4af441ac92b0990518227ea5646b3170b16e6a0e8735d037141

% sudo podman run --detach --name barredis --pod barredis-pod --add-host foo:10.90.0.101 redis
4dff75f4365c66ce892a8bf4f2ee15f57a1bc4d6aea5fb99c6ac0ae2c3e716f6

% sudo podman exec -it barredis cat /etc/hosts
contents of host /etc/hosts
...
10.90.0.7       barredis-pod 0ed9be464360-infra
10.90.0.1 	host.containers.internal
10.90.0.101 	foo
172.16.209.6    host.containers.internal

% sudo podman container rm barredis -f
4dff75f4365c66ce892a8bf4f2ee15f57a1bc4d6aea5fb99c6ac0ae2c3e716f6

% sudo podman run --detach --name barredis --pod barredis-pod --add-host foo:10.90.0.101 redis
77dddb2e845c424c221731f95ee2d857bf6c3976ef312bb0baefb4b6c93a9a42

% sudo podman exec -it barredis cat /etc/hosts
contents of host /etc/hosts
...
10.90.0.7       barredis-pod 0ed9be464360-infra
10.90.0.1 	host.containers.internal
10.90.0.101 	foo
172.16.209.6    host.containers.internal
10.90.0.101 	foo
172.16.209.6    host.containers.internal

In light of this... in addition to ironing out the kinks in /etc/hosts management, I'd still like to lobby for the need for a --hosts option and rethinking how --no-hosts and --add-host interact (at the moment they are mutually exclusive.)

When mounting a hosts file using --volume, "internal" and --add-host entries are not added.

% echo "#EMPTY" > /tmp/emptyhosts
% sudo podman run --detach --name fooredis --add-host foo:10.90.0.100 --volume /tmp/emptyhosts:/etc/hosts:z redis
b13c56fff2f8dcba2ad3369c4d8851bc2c7fab9b2f62b62fe6a9eba1e346e05d
% sudo podman exec -it fooredis cat /etc/hosts
#EMPTY

This means there is no runtime mechanism to both ignore the host's /etc/hosts and get "internal" and additional container host entries. As I illustrated in an earlier comment, I think --no-hosts could be made to work well alongside --add-host and a new --hosts option to facilitate more flexible runtime management of container hosts.

[lovette]

I stumbled across another reason to make hosts management more flexible... the Visual Studio Code Remote-Containers extension fails its container setup if container hosts does not define localhost. I've been running everything with --no-hosts to keep the hosts's hosts out of my containers, which leads to an empty containers hosts file. I've now augmented this to COPY a hosts file with a single localhost entry into the image. This works fine, except as is the current design, --no-hosts prevents podman from adding it's own additional entries to hosts.

[lovette]

I just noticed the activity related to this issue. (GH doesn't send notifications for "commit references".) Will the addition of a system wide base_hosts_file lead towards podman CLI arguments to address the pain points I mentioned?

[Luap99]

Not yet implemented but I see the value, it should be simple to add once my first changes are merged.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 any progress?

[Luap99]

Not yet

[lovette]

What is the current thinking as far as cli arguments go? Is it feasible to enhance --no-hosts to work in concert with --add-host and a new --hosts option to facilitate more flexible runtime management of container hosts? What would a proposed options table based on my 2/28 comment look like, including the influence of base_hosts_file?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 any chance you can work on this?

[lovette]

I've finally had a chance to upgrade podman so my project can take advantage of the new base_hosts_file setting in containers.conf. It's a broad stroke solution but definitely moves podman in the direction I need.

I could achieve nearly all the flexibility I need with the addition of a --base-hosts-file CLI argument that does what base_hosts_file does but at the individual pod/container level.

--base-hosts-file="|none|image|path"

With that in place, the only scenario then missing is the ability to also prevent the "auto" entries from being added to the container /etc/hosts. Essentially what --no-hosts does now but also compatible with --add-host so all three options can be used together.

--base-hosts-file "none" --add-host newhost:1.2.3.4 --no-hosts

[gavinkflam]

I'm adding the CLI flags since I've added a container level base_hosts_file config in #21013.

[lovette]

Thanks @gavinkflam, I'm excited to try this out! Maybe I can use this to work around the fact that --add-host and --no-host can't be used together.