BUG: Requested host port bindings are nondeterministically missing after creating a container in the Go SDK

[mieubrisse]

Description
Via the Go SDK, I'm creating & starting a container with several ports that I want bound to the host, and I don't want to deal with choosing the host ports so I let Docker do it for me. I therefore need to inspect the container after it's created to figure out what host ports the container got. Approximately 10% of the time, I'll inspect the container after creation and NetworkSettings.Ports will have my requested ports as keys but there won't be any values:

&{NetworkSettingsBase:{Bridge: SandboxID:720886a4bed16ebb65140478a77e0c1d2f1a6826817e73c477a3f02a03f3616e HairpinMode:false LinkLocalIPv6Address: LinkLocalIPv6PrefixLen:0 Ports:map[30303/tcp:[] 30303/udp:[] 8545/tcp:[] 8546/tcp:[]] SandboxKey:/var/run/docker/netns/720886a4bed1 SecondaryIPAddresses:[] SecondaryIPv6Addresses:[]} DefaultNetworkSettings:{EndpointID:6a15f7a40bd214d7593bf3b1a8755cf00783ba10495c35e687c234669feeb659 Gateway:172.17.0.1 GlobalIPv6Address: GlobalIPv6PrefixLen:0 IPAddress:172.17.0.12 IPPrefixLen:16 IPv6Gateway: MacAddress:02:42:ac:11:00:0c} Networks:map[KTI2021-09-16T16.55.34-7129:0xc00054e300 bridge:0xc00054e3c0]}

When I do a command-line inspect of the same container immediately afterwards, the values show up:

            "Ports": {
                "30303/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "50165"
                    }
                ],
                "30303/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "62632"
                    }
                ],
                "8545/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "50163"
                    }
                ],
                "8546/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "50164"
                    }
                ]
            },

Steps to reproduce the issue:

1. In the Go SDK, create a nat.PortMap object with some ports to bind, with the value set to []nat.PortBinding{{}} so that Docker will automatically pick the host port
2. Create the container with the HostConfig.PortBindings field set to the port map
3. Immediately afterwards, inspect the running container to try and find the host interface:portnum mappings

Approximately 10% of the time, NetworkSettings.Ports will have keys corresponding to your ports but no values. Doing the same inspect later will work.

Describe the results you received:
NetworkSettings.Ports map has no values

Describe the results you expected:
NetworkSettings.Ports should always have values if the container creation & starting succeeded

Additional information you deem important (e.g. issue happens only occasionally):
This happens nondeterministically, which suggests to me that the Docker Go SDK returns a "success" from starting the container before the host ports are actually bound (which seems like a bug to me).

Output of docker version:

Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:22 2021
 OS/Arch:           darwin/amd64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.4)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 72
  Running: 14
  Paused: 0
  Stopped: 58
 Images: 191
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.844GiB
 Name: docker-desktop
 ID: ULKG:HVAF:XOIN:AZW5:4V7X:6YWU:OCX6:R3VQ:JFRP:TOBR:Z4HE:BA3F
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 192
  Goroutines: 174
  System Time: 2021-09-16T22:15:28.1683327Z
  EventsListeners: 5
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[thaJeztah]

. I therefore need to inspect the container after it's created to figure out what host ports the container got. Approximately 10% of the time, I'll inspect the container after creation

Do you have a minimal code example?

If I'm not mistaken, networking will be attached on container start (not create), and I think it should be synchronous (so after container start, the ephemeral ports should be assigned);

moby/daemon/start.go

Lines 149 to 151 in 9674540

	if err := daemon.initializeNetworking(container); err != nil {
	return err
	}

I see you are running on Docker Desktop; I know there's some rewriting happening on container inspect responses when using Docker Desktop (to make the inspect response match the networking setup needed for Docker Desktop);

Would it be able for you to test your code in a container, and in that container, bind-mount /var/run/docker.sock.raw as the socket to connect with the docker daemon? /var/run/docker.sock.raw allows connecting to the docker engine running in Docker Desktop but without the Docker Desktop proxy (which does the rewriting) in between, so something like;

# mount `/var/run/docker.sock.raw` as `/var/run/docker.sock` inside the container;
docker run -it --rm -v /var/run/docker.sock.raw:/var/run/docker.sock <your container image>