Connections to the port exposed via --publish are dropped and do not reach the contained process

[WhyNotHugo]

Issue Description

Exposing a port via --publish on a system with IPv6 doesn't work.

The default network does not have IPv6 enabled by default, but I enabled it manually.

I edited .local/share/containers/storage/networks/podman.json to include "ipv6_enabled": true, before starting the container to enable IPv6. Connections are now received by podman, but immediately dropped, and never reach the container.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman run --rm --publish 8001:8001 whynothugo/vdirsyncer-devkit-radicale
2. curl 'http://[::1]:8001'

Describe the results you received

*   Trying [::1]:8001...
* Connected to :: (::1) port 8001
> GET / HTTP/1.1
> Host: [::1]:8001
> User-Agent: curl/8.8.0
> Accept: */*
> 
* Request completely sent off
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

Describe the results you expected

*   Trying [::1]:8001...
* Connected to :: (::1) port 8001
> GET / HTTP/1.1
> Host: [::1]:8001
> User-Agent: curl/8.8.0
> Accept: */*
> 
* Request completely sent off
* HTTP 1.0, assume close after body
< HTTP/1.0 302 Found
< Date: Mon, 10 Jun 2024 18:08:56 GMT
< Server: WSGIServer/0.2 CPython/3.8.10
< Location: .web
< Content-Type: text/plain; charset=utf-8
< Content-Length: 18
< 
* Closing connection

podman info output

> podman info
host:
  arch: amd64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-r0
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 97.76
    systemPercent: 1
    userPercent: 1.24
  cpus: 24
  databaseBackend: sqlite
  distribution:
    distribution: alpine
    version: 3.20.0
  eventLogger: file
  freeLocks: 2041
  hostname: hyperion.whynothugo.nl
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.9.1-0-edge
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 11957035008
  memTotal: 67180113920
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-r0
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-r0
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: runc
    package: runc-1.1.12-r3
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      commit: 51d5e94601ceffbbd85688df1c928ecccbfa4685
      spec: 1.0.2-dev
      go: go1.22.3
      libseccomp: 2.5.5
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2024.05.23-r0
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 417h 46m 52.00s (Approximately 17.38 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/hugo/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/hugo/.local/share/containers/storage
  graphRootAllocated: 1930587799552
  graphRootUsed: 1737669390336
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user-1000/containers
  transientStore: false
  volumePath: /home/hugo/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1716231535
  BuiltTime: Mon May 20 20:58:55 2024
  GitCommit: ""
  GoVersion: go1.22.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

Attempting to use localhost instead of a specific IP fail too:

curl -v 'http://localhost:8001'
* Host localhost:8001 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8001...
* Connected to localhost (::1) port 8001
> GET / HTTP/1.1
> Host: localhost:8001
> User-Agent: curl/8.8.0
> Accept: */*
> 
* Request completely sent off
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

I suppose that in some configurations curl might prefer IPv4 and it would work, but that's mostly luck.

[WhyNotHugo]

I edited .local/share/containers/storage/networks/podman.json to include "ipv6_enabled": true, before starting the container to enable IPv6. Connections are now received by podman, but immediately dropped, and never reach the container.

Note that before enabling IPv6 support, the result was the same.

[sbrivio-rh]

Does something like this:

$ podman run --rm --publish 8001:8001 fedora python3 -m http.server -b ::1 8001
::1 - - [10/Jun/2024 22:40:03] "GET / HTTP/1.1" 200 -

$ curl http://[::1]:8001/ >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   865  100   865    0     0   596k      0 --:--:-- --:--:-- --:--:--  844k

work for you?

[Luap99]

First, changing ipv6_enabled to true doesn't do anything unless you actually add a ipv6 subnet to the config, compare a network created with podman network create --ipv6.

Second, as rootless the default (podman) network isn't even used unless you specify --network bridge as the default for rootless since 5.0 is pasta and before that is was slirp4netns.
Pasta only uses ipv6 when the host has "public" ipv6 routes, you can easily test what pasta by running it interactively:

$ pasta --config-net ip a
No interfaces with usable IPv6 routes
Couldn't pick external interface: disabling IPv6
...

[Luap99]

Also as root we do support forward via ::1 at all, see #14491

[WhyNotHugo]

@sbrivio-rh Nope:

> podman run --rm -it --publish 8001:8001 fedora python3 -m http.server -b ::1 8001 
Serving HTTP on ::1 port 8001 (http://[::1]:8001/) ...

> curl 'http://[::1]:8001/'
curl: (7) Failed to connect to ::1 port 8001 after 0 ms: Couldn't connect to server

[WhyNotHugo]

Pasta only uses ipv6 when the host has "public" ipv6 routes, you can easily test what pasta by running it interactively:

I don't really understand how public ips addresses are relevant here; I'm trying to connect to a container running on this same host; no networking is happening across [physical] hosts.

[WhyNotHugo]

Apparently my router was in some bogus state and had no public IPv6. I restarted it and now I have a public IPv6.

The requirements is still a problem: sometimes I visit regions with no IPv6 connectivity, and I still want to run a container on my laptop and connect to it. Actually, sometimes I'm on a train with no public IPv6 or IPv4 at all.

[sbrivio-rh]

I don't really understand how public ips addresses are relevant here; I'm trying to connect to a container running on this same host; no networking is happening across [physical] hosts.

The configuration of the upstream interface is relevant because pasta, by default, tries to mimic as close as possible the host networking. By doing so, in the bigger picture, you can avoid NAT because the container inherits the addresses that are assigned to the upstream interface on the host. See also: #22771 (comment).

Another advantage is that we don't have to hardcode any address or route (like slirp4netns would do), see also https://bugzilla.redhat.com/show_bug.cgi?id=2277954#c5.

This is just the default: you can override the upstream interface with -i, addresses with -a, and so on.

The requirements is still a problem: sometimes I visit regions with no IPv6 connectivity, and I still want to run a container on my laptop and connect to it. Actually, sometimes I'm on a train with no public IPv6 or IPv4 at all.

Right, we realised just recently this isn't great for containers on trains or busses, see: #22737 (reply in thread) and following comments.

I'm currently looking for a viable solution that doesn't break the whole model. The biggest problem I'm facing is that if we skip configuring addresses and routes because none were present on the host (for a given IP version), we risk making issues like #22197 worse: there, it's actually important that pasta refuses to wait until networking is ready (because, in that case, it will be ready, at some point).

The proper solution for that issue would be in systemd (systemd/systemd#3312), but I'm not sure that will ever be addressed, so we can't plan on ignoring that, either.

[WhyNotHugo]

The address on the host can change during the lifetime of the container. If you want to avoid NAT and inherit the same IP on the container, then you're going to have to update the container's IP every time that the host IP changes.

Perhaps it's feasible to assing non-routable IPv6 addresses (if those are the only available) and update the container with routable addresses when/if those are assigned on the host?

In any case, using non-routable addresses would be better than using none, since currently the container is not reachable when using localhost:8000.

[sbrivio-rh]

The address on the host can change during the lifetime of the container. If you want to avoid NAT and inherit the same IP on the container, then you're going to have to update the container's IP every time that the host IP changes.

RIght, that was my idea to start with, but it comes with further complications, see #22737 (reply in thread).

Perhaps it's feasible to assing non-routable IPv6 addresses (if those are the only available) and update the container with routable addresses when/if those are assigned on the host?

That might be a good idea nevertheless, I'll need to check. Patches (tested, in this case ;)) are warmly welcome as well.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Luap99]

Sorry I am not following the conversation here, is there actually a specific work item tracked here in either pasta or podman or can this be closed?

[WhyNotHugo]

@Luap99 Yes, this is still an issue.

To summarise, if the host doesn't have a publicly routable IPv6 address when a container is started, the container cannot be reached from the host (with the default configuration).

[sbrivio-rh]

Sorry I am not following the conversation here, is there actually a specific work item tracked here in either pasta or podman or can this be closed?

Kind of, in the sense that loosening start-up checks and admitting IPv6 addresses that are not routable is one of the bits that could improve support for the use case described at #22737 (reply in thread), in the short term.

To summarise, if the host doesn't have a publicly routable IPv6 address when a container is started, the container cannot be reached from the host (with the default configuration).

...via IPv6, that is.

[WhyNotHugo]

...via IPv6, that is.

This is what Firefox, curl any most other clients try by default. Note that the host is reachable, but refuses the connection, so there is never any reason for clients to retry using IPv4.

[sbrivio-rh]

Note that the host is reachable, but refuses the connection, so there is never any reason for clients to retry using IPv4.

Ouch, I missed this detail.

[Luap99]

This is what Firefox, curl any most other clients try by default. Note that the host is reachable, but refuses the connection, so there is never any reason for clients to retry using IPv4.

I don't understand this part. A connection will always get connection refused when connecting to a local port where nothing is listening. So why should this ever be reason to not retry for curl, firefox, etc...? And trying this locally I see curl and firefox trying ::1 first and then fall back to 127.0.0.1, so what am I missing here?

[sbrivio-rh]

Perhaps it's feasible to assing non-routable IPv6 addresses (if those are the only available) and update the container with routable addresses when/if those are assigned on the host?

That might be a good idea nevertheless, I'll need to check. Patches (tested, in this case ;)) are warmly welcome as well.

I looked into this, but if we include link-local addresses in the set "non-routable" ones we might accept to use (if we don't, that won't fix your use case), it might very well mean that we'll assign, or not, a global unicast address to the guest depending on timing (see also #22197).

Should systemd/systemd#3312 ever get fixed, that would be much less critical and I would go ahead with this kind of change, but as long as it's not, it risks causing bigger issues.

So I'd rather implement a more comprehensive fix that involves monitoring host addresses and routes via netlink. We started reporting some ideas and concerns in section 2. of this pad.

[WhyNotHugo]

I'm not sure that systemd/systemd#3312 would help.

I suppose that you intend to monitor network-online.target. From my experience on laptops with systemd (using iwd), nothing automatically triggers network-online.target; one would need to write some (not-that-trivial) glue code to trigger this unit when a wifi network is connected and has resolved an IP and stop it when the link is down.

Regardless, such a solution would only work on configurations using systemd; the issue would still need a separate portable solution for other distributions.

I'd rather implement a more comprehensive fix that involves monitoring host addresses and routes via netlink.

This sounds a lot more reliable.