Docker Overlay network -service name issue #26480
Comments
|
Can you please provide the full details requested by the issue template? Namely output of Thanks |
|
Same issue @RAJKUMAR49 , on swarm with 3 nodes....
create overlay net1: create 2 services:
container db deployed on node3 ... Can only ping service db from container front-nginx-xxxx on node3 ....
docker version docker info |
|
having exactly the same issue :( |
|
additional info : Hi I am using the Ubuntu 16.04.1 LTS in four hosts. One of the host is Docker swarm master. |
|
I would like add more info... |
|
Docker info: |
|
@ludolac can you able to connect to DB , the web app works using service name of DB ? |
|
update : i can able to connect to Telnet port of MongoDB : |
|
When i access the MongoDB using WebApp , the WebApp giving error : (here both containers are running in 2 different nodes) |
ghost
mentioned this issue
|
@RAJKUMAR49 I think your issue is related to gossip channel not established. Can you attach daemon logs for the node where you are having problems? |
|
@RAJKUMAR49 just curious if you can also to provide the output of docker service inspect of MongoDB and WebApp and more specifically the VIPs parts of it? |
|
came across same issue. +1 to resolved. Urgent need fix it. |
|
anyone able to fix it ? |
|
gossip channel not established, anyone can confirm it? |
any relation between this ticket and the above fix ? |
|
I'm sorry that I don't have any more info for debugging this but: I ran into the same issue just now (docker |
|
@erkie restart the host? are u can retry and confirm it effective |
|
Hi , My Network Configuration in 4 LAB machines are: docker_gwbridge Link encap:Ethernet HWaddr 02:42:0c:92:ea:c0 enp2s0 Link encap:Ethernet HWaddr d4:3d:7e:63:93:d8 lo Link encap:Local Loopback veth9c1d4f4 Link encap:Ethernet HWaddr 62:4d:7b:de:df:de docker_gwbridge Link encap:Ethernet HWaddr 02:42:e3:87:d2:d2 enp2s0 Link encap:Ethernet HWaddr d4:3d:7e:63:93:a0 lo Link encap:Local Loopback veth2cad523 Link encap:Ethernet HWaddr 2e:60:c9:ed:a1:9a Machine 3 : docker_gwbridge Link encap:Ethernet HWaddr 02:42:11:8c:7a:58 enp2s0 Link encap:Ethernet HWaddr d4:3d:7e:63:93:da lo Link encap:Local Loopback veth9815d8c Link encap:Ethernet HWaddr 1e:8f:7a:5d:5b:49 Machine 4 : docker_gwbridge Link encap:Ethernet HWaddr 02:42:64:06:7d:86 enp2s0 Link encap:Ethernet HWaddr 68:05:ca:16:88:29 enp3s0 Link encap:Ethernet HWaddr d4:3d:7e:5e:50:45 lo Link encap:Local Loopback veth1cdb9fb Link encap:Ethernet HWaddr 3a:17:da:c2:f8:27 Do you think above IP address details are correct for Swarm ? |
i have tried to connect Docker Service Ports using telnet. |
|
hi all. 1.12.2-rc1 is release, i have testing above case with latest rc version. it works like a charm. anyone can testing and confirm it fixed again? |
|
hi , |
|
@RAJKUMAR49 the release candidate can be found here; https://github.com/docker/docker/releases/tag/v1.12.2-rc1. You can install it with |
|
I had horrible inconsistencies with 1.12.1 with name resolution. Updated to 1.12.2-rc1 and all my problems went away. A mental note here is that ping only works with containers on the same host while the service exposed port works across hosts. |
|
yeah , docker version 1.12.2 rc1 fixed the overlay network issue. |
|
Thanks for testing! |
|
Facing the same issue with the latest docker 1.12.2 and running in swarm mode.Not able to resolve services by service name. |
|
@bvipparla are your services connected to a custom network? If you don't specify a network, you won't be able to use service discovery by name. |
|
@thaJeztah yes. We've a custom overlay network with 4 nodes running in swarm mode. If we restart all the docker engines, the SD is working fine using the service name. After a while, it doesn't work again. We are building a micro services stack where all of the individual micro-services report about the health to a central discovery server over rabbitmq. We are running the rabbitmq as a swarm service, somehow after a while (like 1-2 hrs or so), the microservices are not able to reach this rabbitmq service and failing to establish channel comms. This is not issue with just the rabbitmq service. I used that as an example. I've searched through the forums about this issue and noticed some issues discussing about the embedded DNS and iptables and VIP's. So played with the swarm services by scaling up and down the services. We have around 12 micro services talking to each other. Shuffled through all of these by scaling up and down randomly and when all the services stabilized, the comms are working fine again. |
|
@bvipparla could you open a new issue with those details (and possibly more details, if relevant)? |
|
@thaJeztah , I can confirm that this issue is happening with Docker versions 1.12.3 and 1.12.4 all running on Ubuntu 16.04 without the fix in systemd network mentioned in #26492 . Here is a print of my swarm cluster:
As you can see, I'm running 4 services: root@ip-10-0-1-100:~# docker service ls
ID NAME REPLICAS IMAGE COMMAND
9vtg1wkhclsh nginx 3/3 nginx
azquqaizlq1c alpine 3/3 alpine sleep 30000
bnamhbjl00az viz 1/1 manomarks/visualizer
brsouv0mqk7u b 1/1 busybox sleep 10000From the ip-10-0-1-100 (manager node), I've run ash through docker exec on one of the containers of alpine service and I see that I could ping only those services that has containers running on the same host. Those services are alpine itself and viz. If I try to ping any other service that doesn't has containers running on the same host, the ICMP packages can't reach those services containers. But the strange behaviour is: If I access the nginx service through tcp port 80, it works, like reported by @ludolac here. But it only showed this behaviour with docker-engine 1.12.3. With 1.12.4, the wget nginx command not worked anymore.... This isssue is preventing me to use Docker 1.12.x with swarm mode in more than one host. IMHO, this is a critical issue in swarm mode. Do I need to open a new thread? Here are a print of the ip-10-0-1-100 (manager node) terminal: root@ip-10-0-1-100:~# docker service ps alpine
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
aoeslwnpf4mruhnae8erf023r alpine.1 alpine ip-10-0-1-100 Running Running 18 minutes ago
3l007wh1l5cxevxj40rdmnvgw alpine.2 alpine ip-10-0-1-100 Running Running 18 minutes ago
e14z2qthcgjtn273jqeugv876 alpine.3 alpine ip-10-0-2-100 Running Running 18 minutes ago
root@ip-10-0-1-100:~# docker exec -ti alpine.1.aoeslwnpf4mruhnae8erf023r ash
/ #
/ #
/ # ping viz
PING viz (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=0.121 ms
^C
--- viz ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.121/0.121/0.121 ms
/ #
/ #
/ # ping nginx
PING nginx (10.0.0.6): 56 data bytes
^C
--- nginx ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss
/ #
/ #
/ # wget nginx
Connecting to nginx (10.0.0.6:80)
index.html 100% |*********************************************************************************************| 612 0:00:00 ETA
/ #
/ #
/ # cat index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
/ #
/ #
/ # rm index.html
/ #
/ #
/ # root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~# apt update
Hit:1 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:4 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Sources [211 kB]
Get:5 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Sources [113 kB]
Get:6 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [3,640 B]
Get:7 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [440 kB]
Get:8 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [370 kB]
Get:9 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [134 kB]
Get:10 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [7,376 B]
Get:11 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:12 https://apt.dockerproject.org/repo ubuntu-xenial InRelease
Get:13 http://security.ubuntu.com/ubuntu xenial-security/main Sources [53.0 kB]
Get:14 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [15.3 kB]
Get:15 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [724 B]
Get:16 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [191 kB]
Get:17 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [78.7 kB]
Get:18 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [64.8 kB]
Get:19 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [35.5 kB]
Get:20 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [2,756 B]
Fetched 2,026 kB in 2s (695 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
8 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@ip-10-0-1-100:~# apt list --upgradable
Listing... Done
apt/xenial-updates,xenial-security 1.2.15ubuntu0.2 amd64 [upgradable from: 1.2.15]
apt-transport-https/xenial-updates,xenial-security 1.2.15ubuntu0.2 amd64 [upgradable from: 1.2.15]
apt-utils/xenial-updates,xenial-security 1.2.15ubuntu0.2 amd64 [upgradable from: 1.2.15]
docker-engine/ubuntu-xenial 1.12.4-0~ubuntu-xenial amd64 [upgradable from: 1.12.3-0~xenial]
libapt-inst2.0/xenial-updates,xenial-security 1.2.15ubuntu0.2 amd64 [upgradable from: 1.2.15]
libapt-pkg5.0/xenial-updates,xenial-security 1.2.15ubuntu0.2 amd64 [upgradable from: 1.2.15]
python3-software-properties/xenial-updates 0.96.20.5 all [upgradable from: 0.96.20.4]
software-properties-common/xenial-updates 0.96.20.5 all [upgradable from: 0.96.20.4]
root@ip-10-0-1-100:~# apt -y dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
apt apt-transport-https apt-utils docker-engine libapt-inst2.0 libapt-pkg5.0 python3-software-properties software-properties-common
8 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 21.4 MB of archives.
After this operation, 214 kB of additional disk space will be used.
Get:1 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libapt-pkg5.0 amd64 1.2.15ubuntu0.2 [702 kB]
Get:2 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libapt-inst2.0 amd64 1.2.15ubuntu0.2 [55.7 kB]
Get:3 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt amd64 1.2.15ubuntu0.2 [1,042 kB]
Get:4 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 docker-engine amd64 1.12.4-0~ubuntu-xenial [19.4 MB]
Get:5 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt-utils amd64 1.2.15ubuntu0.2 [196 kB]
Get:6 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt-transport-https amd64 1.2.15ubuntu0.2 [26.0 kB]
Get:7 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.5 [9,432 B]
Get:8 http://sa-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.5 [19.9 kB]
Fetched 21.4 MB in 0s (38.9 MB/s)
(Reading database ... 70429 files and directories currently installed.)
Preparing to unpack .../libapt-pkg5.0_1.2.15ubuntu0.2_amd64.deb ...
Unpacking libapt-pkg5.0:amd64 (1.2.15ubuntu0.2) over (1.2.15) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
Setting up libapt-pkg5.0:amd64 (1.2.15ubuntu0.2) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
(Reading database ... 70429 files and directories currently installed.)
Preparing to unpack .../libapt-inst2.0_1.2.15ubuntu0.2_amd64.deb ...
Unpacking libapt-inst2.0:amd64 (1.2.15ubuntu0.2) over (1.2.15) ...
Preparing to unpack .../apt_1.2.15ubuntu0.2_amd64.deb ...
Unpacking apt (1.2.15ubuntu0.2) over (1.2.15) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up apt (1.2.15ubuntu0.2) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
(Reading database ... 70429 files and directories currently installed.)
Preparing to unpack .../apt-utils_1.2.15ubuntu0.2_amd64.deb ...
Unpacking apt-utils (1.2.15ubuntu0.2) over (1.2.15) ...
Preparing to unpack .../apt-transport-https_1.2.15ubuntu0.2_amd64.deb ...
Unpacking apt-transport-https (1.2.15ubuntu0.2) over (1.2.15) ...
Preparing to unpack .../docker-engine_1.12.4-0~ubuntu-xenial_amd64.deb ...
Unpacking docker-engine (1.12.4-0~ubuntu-xenial) over (1.12.3-0~xenial) ...
Preparing to unpack .../software-properties-common_0.96.20.5_all.deb ...
Unpacking software-properties-common (0.96.20.5) over (0.96.20.4) ...
Preparing to unpack .../python3-software-properties_0.96.20.5_all.deb ...
Unpacking python3-software-properties (0.96.20.5) over (0.96.20.4) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu12) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for dbus (1.10.6-1ubuntu3.1) ...
Setting up libapt-inst2.0:amd64 (1.2.15ubuntu0.2) ...
Setting up apt-utils (1.2.15ubuntu0.2) ...
Setting up apt-transport-https (1.2.15ubuntu0.2) ...
Setting up docker-engine (1.12.4-0~ubuntu-xenial) ...
Setting up python3-software-properties (0.96.20.5) ...
Setting up software-properties-common (0.96.20.5) ...
Processing triggers for libc-bin (2.23-0ubuntu5) ...
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0413wg8x7p6dkv9ttd42gefye ip-10-0-3-100 Down Active Unreachable
7fzw5kwaj7j3urxapxvsujn44 ip-10-0-22-107 Ready Active
a6z9520td2iw67291igk9qzot ip-10-0-2-100 Ready Active Reachable
cdt5hal5avbwdxiwfw5kymp3k * ip-10-0-1-100 Ready Active Leader
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~# docker service ls
ID NAME REPLICAS IMAGE COMMAND
9vtg1wkhclsh nginx 3/3 nginx
azquqaizlq1c alpine 3/3 alpine sleep 30000
bnamhbjl00az viz 1/1 manomarks/visualizer
brsouv0mqk7u b 1/1 busybox sleep 10000
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~# docker service ps alpine
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
ckqelj734hvx8n8s0eypagmmi alpine.1 alpine ip-10-0-1-100 Running Running about a minute ago
aoeslwnpf4mruhnae8erf023r \_ alpine.1 alpine ip-10-0-1-100 Shutdown Complete about a minute ago
70806xpn57mkr5haiiq20qgsx alpine.2 alpine ip-10-0-1-100 Running Running about a minute ago
3l007wh1l5cxevxj40rdmnvgw \_ alpine.2 alpine ip-10-0-1-100 Shutdown Complete about a minute ago
bpxhm55a9tvhw7t6zrtinjk8a alpine.3 alpine ip-10-0-2-100 Running Running about a minute ago
e14z2qthcgjtn273jqeugv876 \_ alpine.3 alpine ip-10-0-2-100 Shutdown Complete about a minute ago
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~#
root@ip-10-0-1-100:~# docker exec -ti alpine.1.ckqelj734hvx8n8s0eypagmmi ash
/ #
/ #
/ #
/ # ping -c 2 viz
PING viz (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=0.125 ms
64 bytes from 10.0.0.2: seq=1 ttl=64 time=0.083 ms
--- viz ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.083/0.104/0.125 ms
/ #
/ #
/ # ping -c 2 alpine
PING alpine (10.0.0.10): 56 data bytes
64 bytes from 10.0.0.10: seq=0 ttl=64 time=0.036 ms
64 bytes from 10.0.0.10: seq=1 ttl=64 time=0.062 ms
--- alpine ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.036/0.049/0.062 ms
/ #
/ #
/ # ping -c 2 nginx
ping: bad address 'nginx'
/ #
/ #
/ # wget nginx
wget: bad address 'nginx'
/ #
/ #
/ # |
|
@galindro How are your services started? Are they all connected to the same custom network? It's better to open a new issue, because the issue that was reported here was resolved, and there's many possible causes for overlay networking not working (many depending on configuration of the hosts) |
|
@thaJeztah done: #29404 |
I face this issue in Overlay networks. I started all Docker Swarm services in same overlay network using Docker 1.12.1 engine. I can able to access container using service name in the Same host only . Accessing the another host containers using service name is not working . i have even tried the --listen-addr method when launching the Swarm manger and Swarm Worker. Related closed ticket #23855
Also, i can see that Overlay network allocated the IP address to all containers in all the hosts . i can ping the VIP of the service from that service's containers . please help.
The text was updated successfully, but these errors were encountered: