Different port number are getting mapped on ipv4 and ipv6 for same expose port

[gkhandake]

Description

With docker version 20.10.6, build 370c289 observed that different port is assigned for same exposed port on ipv4 and ipv6. This is causing random failures in application with docker compose up which spawns many different apps containers

$ docker ps --format "{{.Ports}}"
0.0.0.0:49338->9999/tcp, :::49337->9999/tcp
0.0.0.0:49339->6379/tcp, :::49338->6379/tcp
0.0.0.0:49319->6379/tcp, :::49318->6379/tcp

Steps to reproduce the issue:

1. Install docker version '20.10.6, build 370c289'
2. use docker-compose up with multiple types of container and expose a port on each container

Describe the results you received:

1. Observe post docker compose up the ipv4 and ipv6 uses different ports

Describe the results you expected:

1. the 1 port should be used for both ipv4 and ipv6 ports for a exposed port

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

20.10.6, build 370c289

Output of docker info:

(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.):

[ncopa]

This is observed on

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:55:27 2021
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker ps shows that the ssh ->22/tcp ports gets different for ipv4 and ipv6.

bcfabeb4ee32   footloose-alpine   "/sbin/init"             23 minutes ago   Up 23 minutes   0.0.0.0:51501->22/tcp, :::51500->22/tcp, 0.0.0.0:51500->6060/tcp, :::51499->6060/tcp, 0.0.0.0:51499->6443/tcp, :::51498->6443/tcp, 0.0.0.0:51498->9443/tcp, :::51497->9443/tcp, 0.0.0.0:51497->10250/tcp, :::51496->10250/tcp   TestBasicSuite-controller0

This seems to happen after docker has been running a while (a few weeks) and/or have many left-over containers.

This happens in an 32 bit armv7 ubuntu lxc container on an aarch64 ubuntu host.

[gkhandake]

Any update on this?

Even a simple docker run after keeping the daemon running for long time we can see this being reproduced:

# docker run --rm -it -p 6650 -p 8080 apachepulsar/pulsar-standalone:2.7.2

On another terminal:

# docker ps
CONTAINER ID   IMAGE                                  COMMAND            CREATED          STATUS          PORTS                                                                                              NAMES
883b78808f76   apachepulsar/pulsar-standalone:2.7.2   "supervisord -n"   24 seconds ago   Up 23 seconds   80/tcp, 0.0.0.0:49686->6650/tcp, :::49685->6650/tcp, 0.0.0.0:49685->8080/tcp, :::49684->8080/tcp   pedantic_zhukovsky

This causes the services to be not accessible e.g.

# curl --verbose http://localhost:49685/admin/v2/brokers/ready
* About to connect() to localhost port 49685 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 49685 (#0)
> GET //admin/v2/brokers/ready HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:49685
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

[gkhandake]

Update.. after daemon restart the ports are correctly assigned.. however we can not restart daemon frequently

# systemctl restart docker
...
# docker run --rm -it -p 6650 -p 8080 apachepulsar/pulsar-standalone:2.7.2

From another terminal

# docker ps
CONTAINER ID   IMAGE                                  COMMAND            CREATED          STATUS          PORTS                                                                                              NAMES
26792898aea6   apachepulsar/pulsar-standalone:2.7.2   "supervisord -n"   24 seconds ago   Up 23 seconds   80/tcp, 0.0.0.0:49158->6650/tcp, :::49158->6650/tcp, 0.0.0.0:49157->8080/tcp, :::49157->8080/tcp   angry_gates

# curl --verbose http://localhost:49157/admin/v2/brokers/ready
* About to connect() to localhost port 49157 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 49157 (#0)
> GET /admin/v2/brokers/ready HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:49157
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 16 Jun 2021 12:52:14 GMT
< broker-address: localhost
< Content-Type: application/json
< Content-Length: 2
< Server: Jetty(9.4.39.v20210325)
< 
* Connection #0 to host localhost left intact
ok

[nathwill]

confirming that we're experiencing this as well, the same host port is getting mapped twice onto different container ports, resulting in randomly broken packet routing.

[dhs-rec]

Same here (Version: 20.10.7-0ubuntu1~20.04.1). Any news?

[jonnybbb]

Is there any workaround available, other than downgrading? Is there any logging available we could provide to you to further investigate the issue?

[f18m]

Hi @thaJeztah ,
I've been reading your comments on #41805.
Do you know if there's any way to ask docker daemon to stop exposing ports over IPv6 and limit it to use only IPv4 ?

That would help a lot as a workaround to this bug (which was created almost 1year ago btw!!)

Thanks!

[allwinchristopher]

We have been frequently encountering the same port mapping issue with latest docker engine. If there are any further updates to this same thread please share us the info.

[n1ngu]

I am starting multiple testcontainers and the same port leads to different services if accessed via 127.0.0.1 or via ::1

This can be disastrous.

Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.18.3
 Git commit:        100c70180f
 Built:             Sat Jun 11 23:27:28 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.3
  Git commit:       a89b84221c
  Built:            Sat Jun 11 23:27:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6.m
 runc:
  Version:          1.1.3
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Contrary to what @ncopa said in #42442 (comment) , this happened on a monday morning to me, so the docker host had less than 2 hours of uptime and there were close to 0 left-over containers.

[kiview]

@n1ngu Can you share which version of Testcontainers are you using when you observe this issue? This seems to be related to dotnet/Docker.DotNet#565 as well.

Furthermore, I wonder if this is a bug or intended behavior and users cannot rely on IPv4 and IPv6 having the same mapping in case of random port publishing?

[n1ngu]

@kiview I was using python's testcontainers package https://pypi.org/project/testcontainers , I will guess it was version 3.6.1 at that time, but I couldn't tell.

While it is clearly technically possible to bind an application to different port numbers on each internet protocol version, I know of no other software that does anything similar. Just as you could also bind to different ports on different IPv4 network interfaces, but you never would.

I guess there is some kind of event that introduces a leap between IP v4 and v6 ports sequences, which were intentionally aligned otherwise. This betrays very basic intuitions on how IP works and I honestly think this a bug beyond debate, not a corner case.

Darn, in a dual stack network this causes a simple hostname:port connection attempt to route to different applications depending on the IPv6 (in)capabilities of both the software and hardware involved in each connection!

[dhs-rec]

@n1ngu Can you share which version of Testcontainers are you using when you observe this issue? This seems to be related to dotnet/Docker.DotNet#565 as well.

No, it's not. It doesn't matter which container is used. We're seeing this on our custom (Linux) containers we use for building. And when it happens, I can try whatever image I want. The problem persists until Docker has been restarted.

[kiview]

Thanks for sharing that you used the Python implementation @n1ngu. That confirms that this is likely no Testcontainers language implementation specific issue (e.g., the bug observed for .NET) but present in all Testcontainers implementations, regarding the assumptions about how Docker maps random ports.

It would be great to get a confirmation from the Docker team whether this is a bug (which I would assume, since very unintuitive and likely caused by some race conditions and misaligned internal state) or it works as intended and we as a client integrating against Docker need to take care of it (our even our users).

[n1ngu]

I came up with some reproduction steps. It is essentially about using the port that docker would pick next but only in the IPv4 interface, and then starting a container with a bound port.

1. Start a container with a bound port

docker run --rm -ti -p 1234 alpine

2. Anotate the port being bound. Now you can stop that container

docker ps
CONTAINER ID   IMAGE     COMMAND     CREATED         STATUS         PORTS                                         NAMES
01ec965c9cb7   alpine    "/bin/sh"   5 seconds ago   Up 4 seconds   0.0.0.0:49191->1234/tcp, :::49191->1234/tcp   gifted_curie

3. Bind another application to the next (N+1) port that would be used by docker, but only for the IPv4 interface. Note that netcat options can vary depending on your actual version. Whatever you do, make sure the IPv6 port is free.

nc -l -p 49192

ss -ntlp

State       Recv-Q      Send-Q                Local Address:Port             Peer Address:Port      Process                                 
LISTEN      0           4                           0.0.0.0:49192                 0.0.0.0:*          users:(("nc",pid=117454,fd=3))         

4. Start another container and check the ports: they mismatch.

docker run --rm -ti -p 1234 alpine
docker ps

CONTAINER ID   IMAGE     COMMAND     CREATED         STATUS         PORTS                                         NAMES
dc8bbe2d4c07   alpine    "/bin/sh"   8 seconds ago   Up 7 seconds   0.0.0.0:49193->1234/tcp, :::49192->1234/tcp   stupefied_visvesvaraya

My guess is that, in either a workstation or in a shared CI agent or in a production docker cluster, any process can bind an arbitrary high port to itself, accidentally triggering this issue if the port is in the sequence that docker expects to be free.

[kiview]

Looping back to this #41805 (comment) from @thaJeztah, shall we assume this behaviour works as intended and there are no guarantees about them being mapped onto the same port?

[n1ngu]

shall we assume this behaviour works as intended and there are no guarantees about them being mapped onto the same port?

No.

Loopback to #42442 (comment)

Darn, in a dual stack network this causes a simple hostname:port connection attempt to route to different applications depending on the IPv6 (in)capabilities of both the software and hardware involved in each connection!

This is unacceptable behaviour. Period.

[dhs-rec]

... since nearly two years now. When will this get fixed?

Since a service restart is required to get correct behaviour again, this is a DoS bug.

[thaJeztah]

As to this being a "DoS" bug; this issue happens when using randomly selected ports from the ephemeral port range; for those, there are no guarantees in any form for those ports to be the same when (re)starting a container or service. If a stable, predictable port is important, then specifying the port-mapping is the correct approach.

As to; assigning the same (random) port for IPv4 and IPv6 mappings; contributions welcome if someone is interested in looking into this.

[m8719-github]

If you run: docker run -p 0.0.0.0::<container-port> it does not map the IPv6 port

[moltar]

This affects GitHub Actions runner as well.

Using these docs:

https://docs.github.com/en/actions/using-containerized-services/about-service-containers#mapping-docker-host-and-service-container-ports

When I specify only a port:

    services:
      postgres:
        image: postgres:11
        ports:
          - 5432

Result is two different ports are created:

  5432/tcp -> 0.0.0.0:32792
  5432/tcp -> :::32791

[martimors]

I came here via testcontainers/testcontainers-dotnet#386 and dotnet/Docker.DotNet#511, and I'm a bit at a loss what to do. Would you say this is a #wontfix on your end, and that the testcontainers project should choose random ports from a lower range, or is this still a bug worth fixing at this level?