docker fails to allocate and map port #10207

LinforPros · 2015-01-20T00:28:22Z

The below message gets generated after:
docker run -it --name apache -d fedora/apache

Failed to allocate and map port 80: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 80 ! -i docker0 -j DNAT --to-destination 172.17.0.8:80: iptables: No chain/target/match by that name.

The system
firewall-cmd --list-all
FedoraServer (default, active)
interfaces: docker0 enp5s0 virbr0
sources:
services: cockpit dhcpv6-client http ssh
ports:
masquerade: no
forward-ports:
icmp-blocks:
rich rules:

jessfraz · 2015-01-20T00:29:47Z

can you give us the output of docker version and docker info

LinforPros · 2015-01-20T01:47:05Z

here it is:

docker info
Containers: 0
Images: 29
Storage Driver: devicemapper
Pool Name: docker-253:1-528723-pool
Pool Blocksize: 65.54 kB
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 1.286 GB
Data Space Total: 107.4 GB
Metadata Space Used: 2.195 MB
Metadata Space Total: 2.147 GB
Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.17.8-300.fc21.x86_64
Operating System: Fedora 21 (Twenty One)
CPUs: 4
Total Memory: 7.698 GiB
Name: localhost.localdomain
ID: U7ER:I52T:BTY6:5WU5:FVLE:PFDT:X3SO:Z6HC:Y5X7:GGIF:Z5JP:VWTG

docker version
Client version: 1.4.0
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 4595d4f/1.4.0
OS/Arch (client): linux/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f/1.4.0

jessfraz · 2015-01-20T01:49:58Z

Ok so as of #7003 being merged,
iptables runles are kept exclusively in a DOCKER chain. I do not think
yours should be failing without, but I would be curious to know if the
binaries at master.dockerproject.com (which have the patch) help your
problem. Otherwise my only guess would be something with Firewalld is
blocking this.

On Mon, Jan 19, 2015 at 5:47 PM, LinforPros notifications@github.com
wrote:

here it is:

docker info
Containers: 0
Images: 29
Storage Driver: devicemapper
Pool Name: docker-253:1-528723-pool
Pool Blocksize: 65.54 kB
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 1.286 GB
Data Space Total: 107.4 GB
Metadata Space Used: 2.195 MB
Metadata Space Total: 2.147 GB
Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.17.8-300.fc21.x86_64
Operating System: Fedora 21 (Twenty One)
CPUs: 4
Total Memory: 7.698 GiB
Name: localhost.localdomain
ID: U7ER:I52T:BTY6:5WU5:FVLE:PFDT:X3SO:Z6HC:Y5X7:GGIF:Z5JP:VWTG

docker version
Client version: 1.4.0
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 4595d4f
4595d4f
/1.4.0
OS/Arch (client): linux/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f
4595d4f
/1.4.0

—
Reply to this email directly or view it on GitHub
#10207 (comment).

LinforPros · 2015-01-20T04:35:40Z

Here is what I get after:
systemctl stop firewalld.service
iptables -L (showing nothing blocked)

Failed to allocate and map port 80: iptables failed: iptables --wait -t nat -
(exit status 1)
iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 80 ! -i docker0 -j DNAT --to-destination 172.17.0.12:80: iptables: No chain/target/match by that
(exit status 1)
time="2015-01-19T22:30:05-06:00" level="info" msg="-job allocate_port(f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac) = ERR (1)"
time="2015-01-19T22:30:05-06:00" level="info" msg="+job release_interface(f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac)"
time="2015-01-19T22:30:05-06:00" level="info" msg="-job release_interface(f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac) = OK (0)"
time="2015-01-19T22:30:05-06:00" level="info" msg="+job release_interface(f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac)"
time="2015-01-19T22:30:05-06:00" level="info" msg="-job release_interface(f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac) = OK (0)"
time="2015-01-19T22:30:05-06:00" level="info" msg="+job log(die, f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac, fedora/apache:latest)"
time="2015-01-19T22:30:05-06:00" level="info" msg="-job log(die, f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac, fedora/apache:latest) = OK (0)"
Cannot start container f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac: (exit status 1)
time="2015-01-19T22:30:05-06:00" level="info" msg="-job start(f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac) = ERR (1)"
time="2015-01-19T22:30:05-06:00" level="error" msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container f5086c123fc9cae55ccb99f09b16f0e0
time="2015-01-19T22:30:05-06:00" level="error" msg="HTTP Error: statusCode=500 Cannot start container f5086c123fc9cae55ccb99f09b16f0e0fa62e298aa5590509e3a3bc08b2fbaac:
lines 958-1000/1000 (END)

What do you make of it?

LinforPros

jessfraz · 2015-01-20T17:58:28Z

Hmmm this is super odd, have you tried adding a rule to the iptables chain manually through the command-line? Does that work? Maybe there is some sort of problem with permissions?

jessfraz · 2015-01-20T18:00:27Z

Also what is the output of iptables-save

jessfraz · 2015-01-20T18:03:26Z

Sorry for all the replies, this is just quite odd.
I have a fedora 21 machine for our jenkins matrix and have not seen this. Maybe also your kernel is missing a specific config. Can you try curling and running this script https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh

p3t · 2015-02-02T16:34:29Z

Hello,
I got exactly the same problem (using centOS 7). I'm not sure what happened but the problem starts this weekend where on Friday everything has worked fine.

The error message

Feb 02 11:32:42 localhost.localdomain docker[4493]: [c5821fc7.allocate_port(84828d377855d7d3deb75b9d34c0b78e32fc0fd0b693d973f58209cafc540d6e)] Received an unexpected error during port allocation: iptables failed: iptables...
Feb 02 11:32:42 localhost.localdomain docker[4493]: (exit status 1)
Feb 02 11:32:42 localhost.localdomain docker[4493]: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 9200 ! -i docker0 -j DNAT --to-destination 10.1.8.98:9200: iptables: No chain/targ...h by that name.
Feb 02 11:32:42 localhost.localdomain docker[4493]: (exit status 1)

System information

[root@localhost ~]# docker info
Containers: 3
Images: 49
Storage Driver: devicemapper
 Pool Name: docker-253:1-25419689-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 1.486 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 2.699 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.84-RHEL7 (2014-03-26)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.4.4.el7.x86_64
Operating System: CentOS Linux 7 (Core)

[root@localhost ~]# docker --version
Docker version 1.3.2, build 39fa2fa/1.3.2

[root@localhost ~]# ./check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config...
info: reading kernel config from /boot/config-3.10.0-123.4.4.el7.x86_64 ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_MACVLAN: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_CGROUP_PERF: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: missing

p3t · 2015-02-02T17:18:29Z

My problem was solved by this hint: #10218
Somehow the docker chain (iptables -t nat -nvL | grep DOCKER) had been deleted.
The reason for that is yet unclear, but I modified some firewall rules maybe this has a connection...

Restarting the docker daemon has restored the chain.

jessfraz · 2015-02-28T01:31:31Z

dup #9047

mikedep333 · 2015-03-04T03:26:16Z

It appears that interacting with Fedora 21's firewalld (such as via the firewall-config GUI) triggers this bug.

This is likely to happen if you need to open up a port for a container.

ljani · 2015-03-18T06:51:00Z

I haven't used firewall-config but firewall-cmd to move docker0 to trusted zone. sudo iptables -t nat -nvL | grep DOCKER returns nothing for me either:

[ljani@fedora]~% sudo iptables -t nat -nvL | grep DOCKER
[ljani@fedora]~%

I'm also on Fedora 21, the Server edition though.

I think firewalld is interfering with raw iptables access and thus causing problems.

There's another problem that --permanently moving docker0 to trusted zone has no effect after reboot, probably because docker0 is created in some different way than "normal" interfaces. So, after each reboot, I have to run firewall-cmd --zone=trusted --change-interface=docker0 to be able to use openshift.

@jfrazelle I think that's a different bug than this one?

EDIT: and yeah, I have that same error in journalctl:

Mar 18 08:39:07 fedora.localdomain docker[1170]: [1be63d1b.allocate_port(8d8259f1df8579666b9452491e52960e9c2ea15593027e3db89fdccfd7c0d984)] Failed to allocate and map port 8484: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8484 ! -i docker0 -j DNAT --to-destination 172.17.0.17:8080: iptables: No chain/target/match by that name.

EDIT2: temporary workaround is to restart Docker (sudo service docker restart) after firewalld has been changed.

mindscratch · 2015-05-27T09:31:04Z

I've run into this issue on CentOS 7 (kernel 3.10.0-123) using Docker 1.6.0 with iptables and firewalld is not running. The workaround is to restart docker, however, in a running system that means containers die which is not acceptable.

soichih · 2015-07-23T02:23:31Z

@ljani

Thanks for this info!

firewall-cmd --zone=trusted --change-interface=docker0

songweian · 2015-10-28T07:22:53Z

@soichih thanks ,i do not know how ,but it works

jessfraz closed this as completed Feb 28, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker fails to allocate and map port #10207

docker fails to allocate and map port #10207

LinforPros commented Jan 20, 2015

jessfraz commented Jan 20, 2015

LinforPros commented Jan 20, 2015

jessfraz commented Jan 20, 2015

LinforPros commented Jan 20, 2015

jessfraz commented Jan 20, 2015

jessfraz commented Jan 20, 2015

jessfraz commented Jan 20, 2015

p3t commented Feb 2, 2015

p3t commented Feb 2, 2015

jessfraz commented Feb 28, 2015

mikedep333 commented Mar 4, 2015

ljani commented Mar 18, 2015

mindscratch commented May 27, 2015

soichih commented Jul 23, 2015

songweian commented Oct 28, 2015

docker fails to allocate and map port #10207

docker fails to allocate and map port #10207

Comments

LinforPros commented Jan 20, 2015

jessfraz commented Jan 20, 2015

LinforPros commented Jan 20, 2015

jessfraz commented Jan 20, 2015

LinforPros commented Jan 20, 2015

jessfraz commented Jan 20, 2015

jessfraz commented Jan 20, 2015

jessfraz commented Jan 20, 2015

p3t commented Feb 2, 2015

The error message

System information

p3t commented Feb 2, 2015

jessfraz commented Feb 28, 2015

mikedep333 commented Mar 4, 2015

ljani commented Mar 18, 2015

mindscratch commented May 27, 2015

soichih commented Jul 23, 2015

songweian commented Oct 28, 2015