Swarm mode does not listen on published ports #32111
Comments
|
The issue can be reliably reproduced using the Cloudformation template from https://gist.github.com/dedalusj/7f0ffd2fb057abee345ff672ce14f867 . The CF will create three manager instances that will form a swarm. You can then ssh into one of the instances and deploy the service from the compose file above using Consistently one of the manager nodes will not expose the port 8080 published by the service. |
|
Yes, i would like to report the same behavior. Port mapping through docker stack with service compose file does not work. docker version: Client: Server: |
|
Looking at the syslog for the instances it seems that docker never calls iptables to setup the listening rules for the ports of the service. I have attached the syslog of the three instances I used for a test where I tried to deploy a service with both traefik and nginx both failing to listen on the published ports. |
|
@dedalusj When node1 joined the swarm, this failure was infact reported: I am thinking this could happen if the daemon, with a swarm config, is restarted too quickly after a stop, because on stop, the cleanup of the sockets could take a little bit. In fact we see the daemon being stopped a second after it became a swarm node: So it looks like this node was first started as a follower ( Can you try to modify your template to insert some delay between these stop/start operations. |
|
Thanks @aboch that was definitely the cause of the problem. After adding a couple of sleep in the setup scripts everything works as expected. |
|
I have a similar problem. Docker in Swarm mode doesn't listen any exposed ports when deploying services to Scaleway C2S server. However everything with exactly the same configuration works perfectly on DigitalOcean hosted servers. No any errors are logged. |
|
This issue should be fixed in 17.05 because of #32283 because now the swarm init/join will be processed only after the swarm leave is complete, after the networking agent has closed the network DB (control plane and its socket). Same after graceful daemon stop/start sequence, stop will wait for the graceful stop of network DB. |
|
Fixed by #32283 But feel free to reopen if you are able to reproduce with 17.05 |
|
I was able to reproduce it with Output of Output of Output of Here is the output of my reproduction of this issue: pi@shpaolin:~/httpd $ docker run -p 80:80 --name httpd -d hypriot/rpi-busybox-httpd
05927cde03f10c2e7c52102a1f610a63949090bc68b120526dc55e73fa6b4907
pi@shpaolin:~/httpd $ curl localhost:80
<html>
<head><title>Pi armed with Docker by Hypriot</title>
<body style="width: 100%; background-color: black;">
<div id="main" style="margin: 100px auto 0 auto; width: 800px;">
<img src="pi_armed_with_docker.jpg" alt="pi armed with docker" style="width: 800px">
</div>
</body>
</html>
pi@shpaolin:~/httpd $ docker stop httpd
httpd
pi@shpaolin:~/httpd $ docker service create -p 80:80 --constraint 'node.role == manager' --detach=true hypriot/rpi-busybox-httpd
988j3cs57b6qc2xeksk7yla6d
pi@shpaolin:~/httpd $ curl localhost:80
curl: (7) Failed to connect to localhost port 80: Connection refused
pi@shpaolin:~/httpd $ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
988j3cs57b6q awesome_jones replicated 0/1 hypriot/rpi-busybox-httpd:latest *:80->80/tcp
pi@shpaolin:~/httpd $ docker service create -p 80:80 --constraint 'node.role == manager' --detach=true hypriot/rpi-busybox-httpd
Error response from daemon: rpc error: code = 3 desc = port '80' is already in use by service 'awesome_jones' (988j3cs57b6qc2xeksk7yla6d)
pi@shpaolin:~/httpd $ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
988j3cs57b6q awesome_jones replicated 0/1 hypriot/rpi-busybox-httpd:latest *:80->80/tcp
pi@shpaolin:~/httpd $ curl localhost:80
curl: (7) Failed to connect to localhost port 80: Connection refused |
|
I'm also able to reproduce it on fedora 26: $ cat /etc/fedora-release $ uname -a $ docker stack deploy -c stack.yml traefik $ docker version Server: $ docker info $ curl localhost:80 $ curl localhost:8080 $ docker service ls $ docker stack ps traefik /Henrik |
|
Same here updated from 17.07 to 17.11 and added a new node, now all services with non host publish are not reachable, only get a connection refused.
|
|
I solved it using an earlier version of "boot2docker". Apparently version 18.09 has problems. docker-machine create -d hyperv --hyperv-virtual-switch "myswitch" --hyperv-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/down I have tested this solution with virtualbox driver (in mac) and with hyper-v (obviously windows) |
|
Same problem with 18.06.2. After rerun |
|
@ushuz how did you get this log? |
|
@treborbrz From syslog, |
Description
I have a swarm cluster composed of 3 managers running on AWS. When I create a new service using a docker compose file none of the managers listen on the published ports for the service.
Steps to reproduce the issue:
docker deploy --compose-file docker-compose.yml traefikfrom one of the managerscurl -v localhost:8080from one of the manager nodesDescribe the results you received:
Describe the results you expected:
Should be able to contact the running service
Additional information you deem important (e.g. issue happens only occasionally):
The compose file for creating the service is:
Output of
docker service inspect --pretty traefik:Output of
sudo netstat -tunap | grep LISTEN:(here I was expecting docker to listen to ports 80 and 8080)
Output of
sudo iptables -nvL -t nat:Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
The docker nodes are running in AWS. The security group attached to the EC2 instances allow all traffic over all protocol between the docker nodes.
The text was updated successfully, but these errors were encountered: