Cannot ping network alias of container where multiple networks exist

[linggao]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
One network is connected on two containers, each with an alias. But pinging the alias failed.

Steps to reproduce the issue:

1. podman network create foo-a
   podman network create foo-b
   podman network create foo-c

2. podman run --name test1 --network foo-a -d alpine sleep 1000
   podman run --name test2 --network foo-c -d alpine sleep 1000

3. podman network connect --alias test1_nw foo-b test1
   podman network connect --alias test2_nw foo-b test2

4. podman exec -it test1 ping test2_nw
   podman exec -it test2 ping test1_nw

Describe the results you received:
#podman exec -it test1 ping test2_nw
ping: bad address 'test2_nw'

#podman exec -it test2 ping test1_nw
ping: bad address 'test1_nw'
WARN[0000] Error resizing exec session 2ef59d91adec9d88a945f24780aabd1ff6e1cb55ea4d93c25e2655098267e9ce: could not open ctl file for terminal resize for container 2af244714cf421896b2d5bbcbc5deb3cf57221e74e90d14992374d7f7622c3ff: open /var/lib/containers/storage/overlay-containers/2af244714cf421896b2d5bbcbc5deb3cf57221e74e90d14992374d7f7622c3ff/userdata/2ef59d91adec9d88a945f24780aabd1ff6e1cb55ea4d93c25e2655098267e9ce/ctl: no such device or address

# podman inspect test1
...
"Networks": {
"foo-a": {
"EndpointID": "",
"Gateway": "10.89.0.1",
"IPAddress": "10.89.0.5",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "d2:7d:20:3a:a0:1e",
"NetworkID": "foo-a",
"DriverOpts": null,
"IPAMConfig": null,
"Links": null
},
"foo-b": {
"EndpointID": "",
"Gateway": "10.89.1.1",
"IPAddress": "10.89.1.3",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "76:8b:b8:87:45:db",
"NetworkID": "foo-b",
"DriverOpts": null,
"IPAMConfig": null,
"Links": null,
"Aliases": [
"test1_nw"
]
}
}
},
...
Please note that the EndpointID is empty for foo-b, while the real docker container has it filled. Maybe this is the source of the problem?

Describe the results you expected:
The two containers test1 and test b are supposed to be able to ping each other through network aliases

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.0.1-dev
API Version:  3.0.0
Go Version:   go1.15.7
Built:        Tue Feb 16 06:47:41 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.0.27-dev, commit: 7310bf13319ee8ed50799b202509bedc27b36cf8'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: lingvs1.dev.edge-fabric.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-240.15.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 5296504832
  memTotal: 8342462464
  ociRuntime:
    name: runc
    package: runc-1.0.0-70.rc92.module+el8.4.0+9980+44630550.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2146758656
  swapTotal: 2146758656
  uptime: 122h 6m 31.06s (Approximately 5.08 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 7
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1613479661
  BuiltTime: Tue Feb 16 06:47:41 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.0.1-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-catatonit-3.0.0-2.module+el8.4.0+9980+44630550.x86_64
podman-3.0.0-2.module+el8.4.0+9980+44630550.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

This is a known limitation of our alias support via dnsname at present - only the first network the container joins will support aliases. So if you create a container in netA but subsequently do a network connect to netB on the same container, only aliases in netA will be accessible to that container. If a network disconnect netA on the same container occurred, I believe that we would begin to use netB aliases.

We are investigating a resolution to this, but it's not an easy problem and could take some time.

@baude What do we want to do with this bug - do we already have a separate issue open / should we close this one as a dupe?

[linggao]

@mheon @baude what we have requested is the function supported on docker. This limitation of podman prevents our product from working with podman and RHEL.
In our case, each container may have one or more dependent containers, and a dependent containers have its own dependent containers. For security reason, a container can only communicate with each child or parent with different networks. So a container may have multiple networks and it communicates with the children and parents with network aliases.
Could you give us a time frame when this function will be supported?

Btw, I disconnect foo-a from container test1 and foo-c from container test2, now both containers have foo-b left. But they still cannot ping each other through network foo-b using alias.

[baude]

if this is an issue on rhel, I encourage you to file a bugzilla for an RFE. Our network stack is not the same as docker's and we make no promises of compatibility in that area. That said, file the RFE and based on our impressions and product management's, we can see what can be done.

[linggao]

@baude thanks for the info. Could you give me the link for the bugzilla?

[baude]

https://bugzilla.redhat.com/

[linggao]

@mheon @baude I do not think podman supports network alias at all.

$podman network create foo-a
$podman run --name test --network foo-a --network-alias test_nw -d alpine sleep 1000
$podman exec -it test ping test_nw
ping: bad address 'test_nw'

Is there any config I missed that caused this problem?
I also tried 2 containers both on foo-a, and got the same result. They cannot ping each other with aliases.

[baude]

you dont exactly provide a lot of information to help with... rpm -q podman-plugins?

[linggao]

rpm -q podman-plugins
package podman-plugins is not installed

[baude]

install it

[linggao]

@baude after I installed podman-plugins the simple cases work for network alias. The multiple network case that I reported originally still does not work. thanks!

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

this should be handled by aardvark in podman 4.0.

[felixsanz]

@baude it isn't. #14262 and containers/aardvark-dns#403