Cannot connect from container to a container

[flixman]

Issue Description

I have ported a project that uses some containers from docker to podman, and they were running successfully with docker. The problem I am encountering is: I have a nginx container that provides access to a container behind it, keycloak. Nginx claims the "keycloak" service name cannot be resolved, but if I get to the nginx container and I nslookup for keycloak, it gets resolved successfully. So... I do not understand what is going on. Is this a know issue of podman, or a problem with the interaction with nginx?

podman version

Client: Podman Engine
Version: 5.0.2
API Version: 5.0.2
Go Version: go1.22.2
Git Commit: 3304dd9-dirty
Built: Thu Apr 18 13:13:19 2024
OS/Arch: linux/amd64

Steps to reproduce the issue

Steps to reproduce the issue

1. start podman: systemctl --user start podman
2. start the services: docker-compose up -d

Describe the results you received

Nginx service fails with [error] 16#16: *86 keycloak could not be resolved (110: Operation timed out), client: 10.89.1.4, server: keycloak.local, request: "GET / HTTP/1.1", host: "keycloak.local:8443". However, should I execute docker-compose exec -it nginx wget http://keycloak:8080 -O - I get the expected output, and I execute docker-compose exec -it nginx nslookup keycloak I get back

Server:         10.89.1.1
Address:        10.89.1.1:53

Non-authoritative answer:
Name:   keycloak.dns.podman
Address: 10.89.1.7

Non-authoritative answer:

Describe the results you expected

nginx can connect to keycloak, as was happening with docker.

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.11-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: e21e7c85b7637e622f21c57675bf1154fc8b1866'
  cpuUtilization:
    idlePercent: 95.16
    systemPercent: 1.93
    userPercent: 2.91
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2034
  hostname: altair
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.9-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 3257126912
  memTotal: 15912157184
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.10.0-2
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: /usr/lib/podman/netavark is owned by netavark 1.10.3-1
    path: /usr/lib/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.15-1
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_04_26.d03c4e2-1
    version: |
      pasta 2024_04_26.d03c4e2
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 1h 17m 38.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 6
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 500856545280
  graphRootUsed: 191252922368
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 54
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.2
  Built: 1713438799
  BuiltTime: Thu Apr 18 13:13:19 2024
  GitCommit: 3304dd95b8978a8346b96b7d43134990609b3b29-dirty
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

I am running on arch linux.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[flixman]

Fixed: Going through nginx configuration I have observed that I had on it resolver 10.89.0.1 valid=30s;, but that the nslookup was coming back using as a resolver 10.89.1.1.

Update. I guess that what happens is: previously I had only this running on podman (so I set the .0.1), then I stopped working on it to switch to another one... that took that .0.1 and today when recreating the containers they have come with the .1.1.