Docker swarm load balancing not working over private network

[agrrh]

Description

Problem is probably similar to #25325. Docker can't reach containers from hostB when I query hostA public address.

I'm using Docker swarm with 2 hosts, they are connected via wireguard tunnel and are reachable to each other. I'm able to ping those hosts from each other using internal addresses.

Then I initialize swarm mode using --advertise-addr, --data-path-addr and --listen-addr options, also stated internal addresses there. Hosts are visible via docker node ls, both active. No errors in syslog.

But when I create service with 2 replicas, I'm facing strange behavior, accessing service via one of public IPs, I'm able to reach only containers which are running on this particular node. Other requests fail with timeout.

Steps to reproduce the issue:

1. Setup wireguard tunnel, check that it works fine.
2. Setup docker in swarm mode.
3. Run a serivce. I'm using this one: agrrh/dummy-service-py. It runs HTTP service on port 80 and answers with container's hostname + random uuid.
4. Scale service at least with 2 replicas. (docker service create --name dummy --replicas 2 --publish 8080:80 agrrh/dummy-service-py)
5. Try to cycle through replicas querying HostA address.

Describe the results you received:

As I said, requests to containers on other nodes fail:

$ http host1:port
{ "hostname": "containerA" } # this container running at host1
$ http host1:port
http: error: Request timed out (30.0s).

$ http host2:port
http: error: Request timed out (30.0s).
$ http host2:port
{ "hostname": "containerB" } # this container running at host2

Describe the results you expected:

I expect to be able to reach all of running containers by querying public address of any single node.

Additional information you deem important (e.g. issue happens only occasionally):

It seems to me that wireguard/tunnel itself is not the cause as I still able to send pings between containers. For example, containerB can reach those containerA addresses:

• 10.255.0.4 @lo ~0.050 ms (looks like this actually don't leave host2)
• 10.255.0.5 @eth0 ~0.700 ms (I can see this with tcpdump on other end, it's reachable!)
• 172.18.0.3 @eth1 ~0.050 ms (this probably don't leave host2 too)

Due to using --advertise-addr I can see packets running between hosts via private interface.

I tried to install ntp and sync the clock but this not helped.

I also attempted to apply various fixes (e.g. turn off masquerading, re-create default bridge with lower MTU, set default bind IP, etc), but got no luck.

I reproduced the issue 3 already times with clean setup and ready to provide collaborators access to my test hosts if you would like to investigate onsite.

Output of docker version:

Same on both hosts:

Client:
 Version:	18.03.0-ce
 API version:	1.37
 Go version:	go1.9.4
 Git commit:	0520e24
 Built:	Wed Mar 21 23:10:01 2018
 OS/Arch:	linux/amd64
 Experimental:	false
 Orchestrator:	swarm

Server:
 Engine:
  Version:	18.03.0-ce
  API version:	1.37 (minimum version 1.12)
  Go version:	go1.9.4
  Git commit:	0520e24
  Built:	Wed Mar 21 23:08:31 2018
  OS/Arch:	linux/amd64
  Experimental:	false

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: rdwi6u922eb93s3z3cq1vuih1
 Is Manager: true
 ClusterID: g8urrtm78sc68oro86k3wvjzf
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.0.5.1
 Manager Addresses:
  10.0.5.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-37-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 481.8MiB
Name: test1
ID: IS5W:2U5W:XDAE:UXIF:KXRR:FQSU:PI7K:UXEQ:OOHK:HC4O:TLZR:P4UU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Wireguard setup guide (assuming you installed it):

### Server

cd /etc/wireguard
umask 077
wg genkey | tee server_private_key | wg pubkey > server_public_key

# /etc/wireguard/wg0.conf 
[Interface]
Address = 10.0.5.1/32
SaveConfig = true
PrivateKey = <paste server private key here>
ListenPort = 51820

[Peer]
PublicKey = <paste client public key here>
AllowedIPs = 10.0.5.2/32

wg-quick up wg0

### Client

cd /etc/wireguard
umask 077
wg genkey | tee server_private_key | wg pubkey > server_public_key

# /etc/wireguard/wg0.conf 
[Interface]
Address = 10.0.5.2/32
PrivateKey = <paste client private key here>

[Peer]
PublicKey = <paste server public key here>
Endpoint = <paste server IP here>:51820
AllowedIPs = 10.0.5.0/24

wg-quick up wg0

Servers should be reachable via internal addresses in a moment after this steps.

[cecchisandrone]

Any update on this? Did you solve the issue in some way?

It happens also to me with 3 nodes swarm on docker 18.06.1-ce

[agrrh]

Not really, I've found a job. 😅

Gonna try to reproduce the issue today and report back.

[agrrh]

Yes, the issue still persists for current wireguard (0.0.20180910-wg1) and docker-ce (18.06.1-ce).

I have 2 nodes, both are are active and reachable over internal addresses but every 2nd request to docker service fails.

Sadly, I stuck at the same point. Could not figure out what blocks requests between docker nodes.

[bagbag]

Same happens for me with wireguard 0.0.20181018 and docker-ce 18.09.0
Have you found a solution for this problem?

[agrrh]

JFYI, found very same issue: #37985

[caguiclajmg]

Issue also seems to manifest on nebula and WireGuard overlays.

Containers can ping each other but for some reason HTTP requests do not receive replies (the target application receives and replies to the request from the load balancer but the load balancer doesn't receive the reply), so I'm guessing this has something to do with swarm's networking.

[mxrlkn]

I just ran into this problem.

I can ping containers across nodes, but I can also do very small requests like hello worlds. Any requests that are slightly bigger in size, just hangs and timeouts.

Running on Tailscale (wireguard), docker 20.10.7/20.10.8, across alpine linux and raspbian nodes.

[stiivo]

I have to bump this issue since im facing the exact same problems. At least and alternative way to connect nodes on 2 private networks would be very helpfull.

[akerouanton]

@stiivo I tried to look into this issue but I'm unable to replicate it with two VMs. Could you share following details please, that would help me (or other contributors) find what's wrong:

• If you use wg-quick, the complete Wireguard config files for both hosts ;
• Otherwise, shell commands used to set up your wg tunnel and the output of wg showconf <interface> ;
• Output of wg show when the bug kicks in ;
• And the output of docker system info & docker version

[stiivo]

Managed to fix it for me.

Since requests to the webserver in my swarm are handed over by the ingress lb but responses are timing out i started testing if it has something to do with the size of the packets which are transferred.
While playing with ping -s $packetsize i realized that packets bigger (if i remember correct) 1420 bytes are dropped.

Wireguard uses a MTU of 1420 and Docker uses a MTU of 1500 per default.

After lowering the MTU of the Ingress Overlay Network everything is working as expected for me.

All steps are done on a manager node:

1. Stop all running services in your swarm
2. docker network rm ingress (remove the ingress overlay network since we are creating a new one with a lower mtu)
3. Create a new ingress overlay network with your prefered subnet & gateway

docker network create --driver overlay --ingress --opt com.docker.network.driver.mtu=1280 --subnet 10.11.0.0/24 --gateway 10.11.0.1 ingress 

4. systemctl restart docker (don't know if i was not patient enough but that populated the new created network to all other nodes)
5. bring back your services

At this point everything was working for me. I'm pretty sure a MTU of 1280 is a bit too low in this case but im still testing.

Hope i could help some other people facing the same problems.

grettings

[JaphyFan]

你好，你的邮件已收到,我会尽快回复。祝你工作顺利，生活愉快！

[caguiclajmg]

Since requests to the webserver in my swarm are handed over by the ingress lb but responses are timing out i started testing if it has something to do with the size of the packets which are transferred. While playing with ping -s $packetsize i realized that packets bigger (if i remember correct) 1420 bytes are dropped.

Wireguard uses a MTU of 1420 and Docker uses a MTU of 1500 per default.

After lowering the MTU of the Ingress Overlay Network everything is working as expected for me.

Mazel tov!

Gave up on this since I posted my reply, but might get back into it if this works for me as well. At one point I thought about MTU but decided it couldn't/wouldn't/shouldn't be. I'll be damned.