telnet tcp port time out with Docker Swarm overlay network #32742

wing731 · 2017-04-20T11:56:33Z

**some containers are able to be pinged from other containers with eth0 ip，but connecting to tcp port times out（includes connecting the container's local tcp port with eth0 ip ) . This only effects some containers but other containers on that same host can communicate.

Additionally, the IP of these containers will not change, whether they are restarted, updated, or migrated
**

Steps to reproduce the issue:

Added 1 master and 2 slave nodes in swarm cluster
Created an overlay network on master node
Run a container on overlay network

Describe the results you received:

-bash-4.1# ip a        
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:c0:a8:00:1a brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.26/20 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.0.23/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fea8:1a/64 scope link 
       valid_lft forever preferred_lft forever
19: eth1@if20: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.3/16 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:3/64 scope link 
       valid_lft forever preferred_lft forever
-bash-4.1# ss -ntl
State      Recv-Q Send-Q                                                      Local Address:Port                                                        Peer Address:Port 
LISTEN     0      128                                                            127.0.0.11:34338                                                                  *:*     
LISTEN     0      50                                                                     :::20880                                                                 :::*     
-bash-4.1# ping  -c 3 192.168.0.26
PING 192.168.0.26 (192.168.0.26) 56(84) bytes of data.
64 bytes from 192.168.0.26: icmp_seq=1 ttl=64 time=0.052 ms
64 bytes from 192.168.0.26: icmp_seq=2 ttl=64 time=0.038 ms
64 bytes from 192.168.0.26: icmp_seq=3 ttl=64 time=0.035 ms

--- 192.168.0.26 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.035/0.041/0.052/0.010 ms

-bash-4.1# telnet 192.168.0.26 20880
Trying 192.168.0.26...
telnet: connect to address 192.168.0.26: Connection timed out

-bash-4.1# telnet 172.18.0.3 20880
Trying 172.18.0.3...
Connected to 172.18.0.3.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

-bash-4.1# telnet 127.0.0.1 20880 
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^[^]
telnet> quit

Describe the results you expected:
I should be able to connect to tcp port with eth0 ip

Additional information you deem important (e.g. issue happens only occasionally):

[
    {
        "Name": "xxx-prd",
        "Id": "7uloy78sm7nx05wms10ia9fs4",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.0.0/20",
                    "Gateway": "192.168.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "698e5c6ede34c50c17a76220460dbf6834d66bc2bf0a353eeffb157c29944c78": {
                "Name": "****t-service-****.1.cr5v2qp7586hucxt6o4pjicci",
                "EndpointID": "22e8c960223b5b4d3827da1cdc3a3fa3ca8082f8d438893b21d499bcdb380f8f",
                "MacAddress": "02:42:c0:a8:00:62",
                "IPv4Address": "192.168.0.98/20",
                "IPv6Address": ""
            },
            "74f69c218ed45a2b7167196974a67a422d7e654ab8c81847a9054fdcb1f22c1f": {
                "Name": "****t-service-****.3.75jcyksaf7g0u7btohd370gjl",
                "EndpointID": "1e250f9c620aa73219e9f375c00a947339ab7d73f2c7e42b586d4d15d341db98",
                "MacAddress": "02:42:c0:a8:00:61",
                "IPv4Address": "192.168.0.97/20",
                "IPv6Address": ""
            },
            "874657fdf16dc45b54723336e8543e567675ad7787e9f83fa18b2fb44872c278": {
                "Name": "****t-service-****.2.4b2k84rsvzuf5yo8cfthnrv33",
                "EndpointID": "7972dd97d046feb9de90d6644495856bcc1c37f36c2df66209ede395743602e7",
                "MacAddress": "02:42:c0:a8:00:83",
                "IPv4Address": "192.168.0.131/20",
                "IPv6Address": ""
            },
            "8d1eae7192261d080a68b2987989776c8397aa731278be437c40b85242433021": {
                "Name": "****s-service-****.3.0gcpcc1zymry1t9kx9c1hgiky",
                "EndpointID": "f261c037f0fc64e66f5c7f601063b420a4218dbb7d5203fcd2e0122f91dc4db2",
                "MacAddress": "02:42:c0:a8:00:1a",
                "IPv4Address": "192.168.0.26/20",
                "IPv6Address": ""
            },
            "c49fa540f9fc000c990da2e0ca8e4813b9b85469b92a1b0b70e155d740b2f803": {
                "Name": "****f-service-****.1.9doc3x1immdgvdv7kd5eg7snj",
                "EndpointID": "978f7e3dc7ab98c02fd1c655acee0c1a2eff849c38fe0c6a15e48fa75a01af12",
                "MacAddress": "02:42:c0:a8:00:97",
                "IPv4Address": "192.168.0.151/20",
                "IPv6Address": ""
            },
            "eb6225038e26f9636b331b4aa5813259750d87daa53c4a7a506cbae7a45a866c": {
                "Name": "****t-service-****.2.adsdo0iuogmub62lo1jqsbysv",
                "EndpointID": "37ec10dd941290a9f10b9fbce07a6dd387b8280c0ffbb14ed3abc29fd05c0626",
                "MacAddress": "02:42:c0:a8:00:50",
                "IPv4Address": "192.168.0.80/20",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": {}
    }
]

Output of docker version:

Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 5
 Running: 5
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.12.3
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: active
 NodeID: 6jolkl6lcxisee738a6b3vq3m
 Is Manager: false
 Node Address: 10.8.37.214
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.26 GiB
Name: ******-common-prd-docker-010.*****
ID: XZPL:KPA7:CZAB:HDCC:5ZMS:PKCQ:TYGZ:B5CI:GOG5:45SN:MM6R:RABW
Docker Root Dir: /data/docker/images
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 81
 Goroutines: 133
 System Time: 2017-04-20T19:48:30.627676133+08:00
 EventsListeners: 5
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 docker.16qian.cn:5000
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
KVM

The text was updated successfully, but these errors were encountered:

wing731 · 2017-04-21T07:14:30Z

-bash-4.1# telnet 192.168.0.26 20880
Trying 192.168.0.26...
telnet: connect to address 192.168.0.26: Connection timed out

tcpdump

Frame 70: 96 bytes on wire (768 bits), 96 bytes captured (768 bits)
    Encapsulation type: Linux cooked-mode capture (25)
    Arrival Time: Apr 21, 2017 14:19:02.233726000 
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1492755542.233726000 seconds
    [Time delta from previous captured frame: 0.108502000 seconds]
    [Time delta from previous displayed frame: 0.108502000 seconds]
    [Time since reference or first frame: 17.934014000 seconds] <======= 
    Frame Number: 70
    Frame Length: 96 bytes (768 bits)
    Capture Length: 96 bytes (768 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:icmp:ip:tcp]
    [Coloring Rule Name: ICMP errors]   < <======= ERROR
    [Coloring Rule String: icmp.type eq 3 || icmp.type eq 4 || icmp.type eq 5 || icmp.type eq 11 || icmpv6.type eq 1 || icmpv6.type eq 2 || icmpv6.type eq 3 || icmpv6.type eq 4]
Linux cooked capture
    Packet type: Unicast to us (0)
    Link-layer address type: 772
    Link-layer address length: 6
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.0.26, Dst: 192.168.0.26
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 80
    Identification: 0x3af4 (15092)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: ICMP (1)
    Header checksum: 0xbd74 [validation disabled]
    [Header checksum status: Unverified]
    Source: 192.168.0.26
    Destination: 192.168.0.26
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 1 (Host unreachable)
    Checksum: 0x511c [correct]
    [Checksum Status: Good]
    Unused: 00000000
    Internet Protocol Version 4, Src: 192.168.0.26, Dst: 192.168.0.26
        0100 .... = Version: 4
        .... 0101 = Header Length: 20 bytes (5)
        Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
            0000 00.. = Differentiated Services Codepoint: Default (0)
            .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
        Total Length: 52
        Identification: 0x9aab (39595)
        Flags: 0x02 (Don't Fragment)
            0... .... = Reserved bit: Not set
            .1.. .... = Don't fragment: Set
            ..0. .... = More fragments: Not set
        Fragment offset: 0
        Time to live: 64
        Protocol: TCP (6)
        Header checksum: 0x1e94 [validation disabled]
        [Header checksum status: Unverified]
        Source: 192.168.0.26
        Destination: 192.168.0.26
        [Source GeoIP: Unknown]
        [Destination GeoIP: Unknown]
    Transmission Control Protocol, Src Port: 32958, Dst Port: 20880, Seq: 2713485458
        Source Port: 32958
        Destination Port: 20880
        Sequence number: 2713485458
        [Stream index: 14]
        Sequence number: 2713485458    (relative sequence number)
        Acknowledgment number: 0
        Header Length: 32 bytes
        Flags: 0x002 (SYN)
            000. .... .... = Reserved: Not set
            ...0 .... .... = Nonce: Not set
            .... 0... .... = Congestion Window Reduced (CWR): Not set
            .... .0.. .... = ECN-Echo: Not set
            .... ..0. .... = Urgent: Not set
            .... ...0 .... = Acknowledgment: Not set
            .... .... 0... = Push: Not set
            .... .... .0.. = Reset: Not set
            .... .... ..1. = Syn: Set
                [Expert Info (Chat/Sequence): Connection establish request (SYN): server port 20880]
                    [Connection establish request (SYN): server port 20880]
                    [Severity level: Chat]
                    [Group: Sequence]
            .... .... ...0 = Fin: Not set
            [TCP Flags: ··········S·]
        Window size value: 43690
        [Calculated window size: 43690]
        Checksum: 0x81ac [unverified]
        [Checksum Status: Unverified]
        Urgent pointer: 0
        Options: (12 bytes), Maximum segment size, No-Operation (NOP), No-Operation (NOP), SACK permitted, No-Operation (NOP), Window scale
            Maximum segment size: 65495 bytes
                Kind: Maximum Segment Size (2)
                Length: 4
                MSS Value: 65495
            No-Operation (NOP)
                Type: 1
                    0... .... = Copy on fragmentation: No
                    .00. .... = Class: Control (0)
                    ...0 0001 = Number: No-Operation (NOP) (1)
            No-Operation (NOP)
                Type: 1
                    0... .... = Copy on fragmentation: No
                    .00. .... = Class: Control (0)
                    ...0 0001 = Number: No-Operation (NOP) (1)
            TCP SACK Permitted Option: True
                Kind: SACK Permitted (4)
                Length: 2
            No-Operation (NOP)
                Type: 1
                    0... .... = Copy on fragmentation: No
                    .00. .... = Class: Control (0)
                    ...0 0001 = Number: No-Operation (NOP) (1)
            Window scale: 9 (multiply by 512)
                Kind: Window Scale (3)
                Length: 3
                Shift count: 9
                [Multiplier: 512]

allencloud · 2017-05-08T07:46:34Z

@wing731 To be honest, I cannot understand this quite well. eth0 is a host network interface or container interface?

In addition, here you mentioned that:

2.Created an overlay network on master node
3.Run a container on overlay network

Does you mean that in your swarm cluster(created by using docker swarm init) , you use docker run to add a container into an overlay network?

If that, I think Add --attachable network support to enable docker run to work in swarm-mode overlay network is only supported in docker 1.13.0+ (while your environment is 1.12.3) with pr #25962. And this information is described in CHANGELOG.md.

Please correct me if I misunderstood your thought. And if you missed something, please add more details in the issue description. Thanks a lot 😸

wing731 · 2017-05-08T11:57:29Z

@allencloud eth0 is container interface.

This network problem only effects some containers.

After debug, we found have 3 exception IP addresses, and when these IP addresses are allocated to containers, the network of these containers are not available.

But other containers on that same host can communicate.

IP list: 192.168.0.26/20, 192.168.0.90/20, 192.168.0.98/20

thaJeztah · 2023-09-18T12:22:53Z

Let me close this ticket for now, as it looks like it went stale.

GordonTheTurtle added area/networking area/swarm version/1.12 labels Apr 20, 2017

thaJeztah closed this as not planned Won't fix, can't repro, duplicate, stale Sep 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

telnet tcp port time out with Docker Swarm overlay network #32742

telnet tcp port time out with Docker Swarm overlay network #32742

wing731 commented Apr 20, 2017 •

edited

wing731 commented Apr 21, 2017

allencloud commented May 8, 2017 •

edited

wing731 commented May 8, 2017

thaJeztah commented Sep 18, 2023

telnet tcp port time out with Docker Swarm overlay network #32742

telnet tcp port time out with Docker Swarm overlay network #32742

Comments

wing731 commented Apr 20, 2017 • edited

wing731 commented Apr 21, 2017

allencloud commented May 8, 2017 • edited

wing731 commented May 8, 2017

thaJeztah commented Sep 18, 2023

wing731 commented Apr 20, 2017 •

edited

allencloud commented May 8, 2017 •

edited