podman differs with docker in ipv6 functionality

[ajakk]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

With both docker and podman, I can run a registry container that listens on port 5000, something like:

docker run -d -p 5000:5000 registry:latest
or
podman run -d -p 5000:5000 registry:latest

With docker, this results in the container being accessible over ipv6:

# curl -I http://[::1]:5000
HTTP/1.1 200 OK
Cache-Control: no-cache
Date: Sun, 05 Jun 2022 23:42:59 GMT

With podman, this is not the case:

# curl -I http://[::1]:5000
curl: (7) Failed to connect to ::1 port 5000 after 0 ms: Connection refused

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.2.0-dev
API Version:  4.2.0-dev
Go Version:   go1.18.2
Git Commit:   eb26fa45f1326191dea27f2afabf82cb8b934140
Built:        Sun Jun  5 15:05:18 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: app-containers/conmon-2.1.0
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.0, commit: v2.1.0'
  cpuUtilization:
    idlePercent: 93.99
    systemPercent: 2.77
    userPercent: 3.24
  cpus: 8
  distribution:
    distribution: gentoo
    version: "2.8"
  eventLogger: journald
  hostname: sol
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.12-gentoo-dist-hardened
  linkmode: dynamic
  logDriver: journald
  memFree: 1143242752
  memTotal: 14641152000
  networkBackend: cni
  ociRuntime:
    name: crun
    package: app-containers/crun-1.4.5
    path: /usr/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.2.0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 6807629824
  swapTotal: 8589930496
  uptime: 94h 48m 10.45s (Approximately 3.92 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 377315987456
  graphRootUsed: 288568082432
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1652017079
  BuiltTime: Sun May  8 08:37:59 2022
  GitCommit: e4b03902052294d4f342a185bb54702ed5bed8b1
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

N/A - reproducible from git

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes.

Additional environment details (AWS, VirtualBox, physical, etc.):

This is on Gentoo on a physical system.

[mheon]

@Luap99 Interested in this one?

[Luap99]

Do you run podman as root? I don't really know about docker but AFAIK they also do not have ipv6 enabled by default.

As rootless it should just work with podman v4.0 or newer.
For rootful you would need to add a ipv6 subnet to the default cni/netavark config but forwarding via ::1 will not work due a kernel limitation.

[ajakk]

Yes, I'm only testing as root. However, I've done no explicit ipv6 configuration to either podman or docker.

# docker network inspect bridge  | grep -i ipv6
        "EnableIPv6": false,
# podman network inspect podman  | grep -i ipv6
          "ipv6_enabled": false,

[Luap99]

Looks like the docker-proxy process takes care of the port forwarding when no iptables rule match.

[Luap99]

So we could do the same thing as docker and in addition to the iptables rules create an extra user space proxy process.
We already have such a process for rootless called rootlessport, so all we would have to do to use for root as well and maybe rename it.

The advantage is that this would allow us ::1 port forwarding. However this would spawn an extra process per container which consumes extra memory and cpu. I guess we should add an containers.conf option in case this is not wanted.

Also rootlessport does not have sctp support but I guess this is not really important. At least I am not aware of any sctp users.

@mheon WDYT?

[mheon]

@Luap99 Part of the original design discussion for aardvark was to implement this there, given that it's already a running process so (theoretically) the overhead of also forwarding traffic should not be high, but that would require us reimplementing parts of rootlessport in Rust, which does seem undesirable. Never thought of reusing the existing code for this; it sounds technically attractive, though I'd definitely want it to be optional.

[Luap99]

Aardvark does not sounds like a good place for me, IMO it should only do dns. Itwill not be spawned if the network has no dns. Also we would need to spawn many threads to join different net namespaces which will get messy.

It would make more sense to implement this in conmon-rs but I don't know the time line for that.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 any movement on this one?

[Luap99]

No this is a lot of work

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ajakk]

Still an issue.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.