In Swarm, overlay network IP address are not matching with the ip address found inside other service.

[dewang-agrawal]

I have 4 services from which 3 service are running on dockervm1 and one service is running on dockervm5.Below are the information:

docker info:

Containers: 24
 Running: 21
 Paused: 0
 Stopped: 3
Images: 293
Server Version: 1.13.0
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: 6z2pofbqed5tbdoz3y9q72fv6
 Is Manager: true
 ClusterID: 5pk2zxx8dxwxpzya7l4q0vyok
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.44.XXX.XX
 Manager Addresses:
  10.44.XXX.XX:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.51 GiB
Name: dockervm1
ID: W7GW:P6LB:3V4Z:EJ6W:2YTF:W6VX:K5UB:7TZR:AVOI:SLEH:PKNK:S55O
Docker Root Dir: /xfs100
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: true
Insecure Registries:
 dockervm1.persistent.co.in:5000
 127.0.0.0/8
Live Restore Enabled: false

docker version:

Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Wed Jan 18 16:20:26 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Wed Jan 18 16:20:26 2017
 OS/Arch:      linux/amd64
 Experimental: true

docker-compose :

version: "3"
services:
   db_svr:
    image: docker/db_svr
    command: ["db2start"]
    environment:
      - DB2INST1_PASSWORD=dbserver
      - LICENSE=accept
    ports:
      - "50001:50000" 
    volumes:
      - /xfs30/mnt_files/db2/logs:/dblogs
      - /xfs30/mnt_files/db2/mnt:/home/db2inst1/db2inst1
      - /xfs30/mnt_files/db_shared_file.txt:/shared_file.txt
   mqm_svr:
    image: docker/mqm_svr_2
    environment:
      - LICENSE=accept 
      - MQ_QMGR_NAME=ALL1.CORE.MQM
    ports:
      - "35491:1414"
    volumes:
      - /xfs30/mnt_files/mq/mqm:/var/mqm
      - /xfs30/mnt_files/mq_shared_file.txt:/shared_file.txt
   app1:
    image: app1
    command: [ "1" ]
    ports:
      - 8000:4000
    volumes:
      - /xfs30/NewDB/db2/java:/opt/component/java
      - /home/core_pwd.txt:/core_pwd.txt
      - /xfs30/mnt_files/logs:/logs
      - /xfs30/mnt_files/journals:/journals
      - /xfs30/mnt_files/db_shared_file.txt:/db_shared_file.txt
      - /xfs30/mnt_files/mq_shared_file.txt:/mq_shared_file.txt
    environment:
      - REFERENCE_DATE=$REFERENCE_DATE
      - VIRTUAL_DATE=$VIRTUAL_DATE
      - CLASSPATH_PREFIX=/opt/component/lib/*:/opt/component/java/*
      - JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=4000
    deploy:
       placement:
         constraints:
            - node.hostname == dockervm1

   app2:
    image: app2
    ports:
      -  $DEBUG_PORT:4000
    volumes:
      - /xfs30/NewDB/db2/java:/opt/component/java
      - /home/core_pwd.txt:/core_pwd.txt
      - /xfs30/mnt_files/logs:/logs
      - /xfs30/mnt_files/journals:/journals
      - /xfs30/mnt_files/db_shared_file.txt:/db_shared_file.txt
      - /xfs30/mnt_files/mq_shared_file.txt:/mq_shared_file.txt
    environment:
      - REFERENCE_DATE=$REFERENCE_DATE
      - VIRTUAL_DATE=$VIRTUAL_DATE
      - CLASSPATH_PREFIX=/opt/component/lib/*:/opt/component/java/*
      - JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=4000
    command: [ "1" ]
    deploy:
       placement:
         constraints:
            - node.hostname == dockervm5  
networks:
  network_test:
    driver: overlay

After deploying the above docker-compose, 4 services are created from which one service is deployed on dockervm5 and 3 service's are deployed on dockervm1.

While pinging mqm_svr from app2 which is running on dockervm5 the ip address it picks up is 10.0.0.4. But ip address of the mqm_svr is 10.0.0.5 as shown below.

docker network inspect shows ip address of mqm_svr as 10.0.0.5:

[dockervm1 docker_poc]$ docker network inspect e2_default
[
    {
        "Name": "e2_default",
        "Id": "qrdoy4w65xkrph97va8nba8ws",
        "Created": "2017-08-17T10:50:32.656650889-04:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "561958918bf5933bc09db48585fd4f1b64c9c4a552900133f73c7fc298700839": {
                "Name": "e2_mqm_svr.1.mnr49u0uisueqv0ehjd3b8882",
                "EndpointID": "f11c5dc2f671dcddf617f3a9899a63342fcf8a728c3b4dc041c6efd33a2875ca",
                "MacAddress": "02:42:0a:00:00:05",
                "IPv4Address": "**10.0.0.5/24**",
                "IPv6Address": ""
            },
            "966e94dd9a476304d70cf608dcf7996128420c714865b16c8257a360fc25c46d": {
                "Name": "e2_db_svr.1.oit5vosri29r30b8p3pt52dav",
                "EndpointID": "2e8692860c0f8d852fa44158aa72e08de651b233a2096cc183a4d59bc4a732ec",
                "MacAddress": "02:42:0a:00:00:03",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            },
            "e0c2b4deadb4c011f653f85474d5c235c1a8ce689fda2338475c31aff79c8901": {
                "Name": "e2_app1.1.sfb94yr1kq3l8ag3jxsovj0zo",
                "EndpointID": "85d022ec23a7abad9e8ecbc8c532401f1e0f99e1664d66a6bed22910ae5fe2d2",
                "MacAddress": "02:42:0a:00:00:07",
                "IPv4Address": "10.0.0.7/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {
            "com.docker.stack.namespace": "e2"
        },
        "Peers": [
            {
                "Name": "dockervm1-a6af9aae10cd",
                "IP": "10.XX.XXX.XX"
            },
            {
                "Name": "dockervm5-0a060291c137",
                "IP": "10.XX.XXX.XX"
            }
        ]
    }
]

This is the ping command executed from app2 which is running on dockervm5 which is showing ip address of mqm_svr as 10.0.0.4 same is happening with db_svr.
Also same behaviour is observed from app1 which is running on dockervm1.

bash-4.3# ping mqm_svr
PING mqm_svr (10.0.0.4): 56 data bytes
64 bytes from 10.0.0.4: seq=0 ttl=64 time=0.119 ms
64 bytes from 10.0.0.4: seq=1 ttl=64 time=0.105 ms
64 bytes from 10.0.0.4: seq=2 ttl=64 time=0.132 ms
64 bytes from 10.0.0.4: seq=3 ttl=64 time=0.120 ms
^C
--- mqm_svr ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.105/0.119/0.132 ms

Also app1 is able to connect to mqm_svr but app2 is not able to connect to mqm_svr which is running on dockervm5.

Can anyone please help?

[thaJeztah]

The IP address you see in network inspect is the IP-address for tasks backing the service, this can be a single task (container) or multiple.

When you ping mqm_svr, you're pinging the service, and you'll get the services VIP;

• if you ping the task itself (e2_mqm_svr.1.mnr49u0uisueqv0ehjd3b8882), do you get the correct IP?
• if you docker service inspect --format '{{json .Endpoint.VirtualIPs}}' e2_mqm_svr, what IP address does it show?

[dewang-agrawal]

• When I ping the task (e2_mqm_svr.1.mnr49u0uisueqv0ehjd3b8882), I get the correct IP address (10.0.0.5). But my packets are getting lost.
• And when I do docker service inspect --format '{{json .Endpoint.VirtualIPs}}' e2_mqm_svr, I get service VIP i.e. 10.0.0.4.

[yashgt]

If there are 2 services app1 and app2, configured to have a single container each, when the app1 process running in the container project_app1_1 wants to talk to app2 on port 5000, should it talk on project_app2_2:5000 or app2:5000?

Also, I do not see the app2 entry added in the /etc/hosts of the container for app1. In such a case, what does docker use as the DNS?

[thaJeztah]

the full name of the service is project_app1, but an alias without the project_ prefix is also created when deploying a stack.

So other services in the stack can use app1:5000 to connect.

Also, I do not see the app2 entry added in the /etc/hosts of the container for app1. In such a case, what does docker use as the DNS?

Docker uses an embedded DNS server that listens on 127.0.0.11, you'll see that configured as DNS in /etc/resolv.conf inside each container.

[yashgt]

So from the original message on this thread, mqm_svr:1414 should be accessible to app1. We tried doing a telnet to mqm_svr:1414 and got a connection timeout.

[dewang-agrawal]

mqm_svr:1414 and app1 are running on dockervm1 and app2 is running on dockervm5. When I telnet to mqm_svr from app2 to 1414 port, it says connection timeout. But when i try to telnet to mqm_svr from app1 which is also running on dockervm1 , it gets connected.
Telnet should be possible from both app1 and app2 whether they are running on dockervm1 or dockervm5. Can you please highlight any issue here?

[dewang-agrawal]

Can anyone please point me to some reference or any other issue related to this.