Port mapping behaviour differs between docker-machine and centos 7 - the mapped host port is open even though the container port is closed

[CRogers]

BUG REPORT INFORMATION

List the steps to reproduce the issue:

On OS X, using docker-machine 0.5.0 and docker 1.9.0:

$ docker run -d -p 999:999 busybox sleep 100000
ecfb24ef7eed857548babb86974eb5444a72acdf777edb393effb61bf9cd5843

$ docker ps
CONTAINER ID     IMAGE      COMMAND           CREATED          STATUS          PORTS                    NAMES
ecfb24ef7eed     busybox    "sleep 100000"    5 seconds ago    Up 4 seconds    0.0.0.0:999->999/tcp     sharp_kilby

$ nmap -A $(docker-machine ip default) -p 999
Starting Nmap 6.47 ( http://nmap.org ) at 2015-11-24 10:49 GMT
Nmap scan report for 192.168.99.100
Host is up (0.00037s latency).
PORT    STATE  SERVICE VERSION
999/tcp closed garcon

On Centos 7, with docker 1.9.1:

$ docker run -d -p 999:999 busybox sleep 100000
ea4f5982286c2a0a0c06687be94d3913020500f145c4e607e1c32938405f6a5d

$ docker ps
CONTAINER ID     IMAGE      COMMAND          CREATED          STATUS          PORTS                   NAMES
ea4f5982286c     busybox    "sleep 100000"   3 seconds ago    Up 2 seconds    0.0.0.0:999->999/tcp    determined_roentgen

$ nmap -A localhost -p 999
Starting Nmap 6.40 ( http://nmap.org ) at 2015-11-24 10:52 GMT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000038s latency).
Other addresses for localhost (not scanned): 127.0.0.1
PORT    STATE SERVICE    VERSION
999/tcp open  tcpwrapped

Describe the results you received:

• On centos 7, the mapped host port appears open and you can connect to it. You can also send data to it.
• On docker-machine on OS X has the mapped host port appears closed and you cannot connect to it.

Describe the results you expected:

• On both platforms the port mapped on the host is closed, as the container port is closed/does not exist. You cannot connect to the host port.

Provide additional info you think is important:

This is problematic for us as we are trying to wait for services to start from outside the docker container without knowing if they are http, https, other protocol etc. It works fine with docker-machine, but with centos 7 we cannot simply try to connect to the socket, as it will connect.

Another problem is we get really odd errors when pinging these services if they aren't up; for example performing a HTTPS get will result in a SSLHandshakeException rather than a ConnectionRefused.

---

More OS X information:

$ docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      linux/amd64

$ docker info
Containers: 4
Images: 383
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 391
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.12-boot2docker
Operating System: Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov  3 19:49:22 UTC 2015
CPUs: 8
Total Memory: 3.858 GiB
Name: default
ID: YUGC:SSDN:BXRU:BNL7:KIYL:JNGI:H2MW:RJIW:BXDZ:VXVU:AKNQ:MMYZ
Debug mode (server): true
 File Descriptors: 30
 Goroutines: 56
 System Time: 2015-11-24T11:07:54.487607621Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Username: <redacted>
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox

$ uname -a
Darwin <redacted hostname> 14.5.0 Darwin Kernel Version 14.5.0: Tue Sep  1 21:23:09 PDT 2015; root:xnu-2782.50.1~1/RELEASE_X86_64 x86_64

More Centos 7 infomation:

$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

$ docker info
Containers: 9
Images: 26
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-202:3-75497612-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 495.5 MB
 Data Space Total: 107.4 GB
 Data Space Available: 49.72 GB
 Metadata Space Used: 1.561 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.20.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 1
Total Memory: 992.1 MiB
Name: <redacted hostname>
ID: OME7:SHMY:75BZ:L74G:ZUIY:TMJX:3OC7:KSQG:X4WH:WGIZ:MES6:UWV4

$ uname -a
Linux <redacted hostname> 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

----------END REPORT ---------

[GordonTheTurtle]

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

[CRogers]

Edited the issue with more information.

[choliver]

To clarify the "can send data" is by doing echo "hello" | nc 127.0.0.1 -p 999. We get the following TCP sequence:

• -> SYN
• <- SYN/ACK
• -> ACK
• <- FIN/ACK
• -> PSH/ACK ("Hello")
• <- RST

[cpuguy83]

This is due to the userland proxy which handles local connections. You can disable it on the daemon with --userland-proxy=false.
But really, I'd suggest connecting to the container's IP/port directly.

[CRogers]

Thanks for the response! How do I run with userland proxy off/change it to be off? I see it's under the docker daemon command, but how do I use that to turn it off for a currently running docker?

[cpuguy83]

@CRogers You have to restart docker.