TCP SYN sent to all replicas from a service on an ingress network

[networkingana]

I have a service with 3 replicas waiting for requests on published port (in my case 1402) and my goal is to load balance the incoming requests for this service.

I created two networks like following

docker network create --driver overlay --subnet=172.18.1.0/27 --attachable net1
docker network create --driver overlay --subnet=172.18.0.0/27 --ingress net2 

and I removed the existing network

My service has two virtual IPs 172.18.0.7 and 172.18.1.9. And each of the 3 containers has:
replica1: 172.18.0.8, 172.18.1.10
replica2: 172.18.0.9, 172.18.1.11
replica3: 172.18.0.10 172.18.1.12

I expected when the incoming request reaches the VIP of the service it will be sent only to one of the containers. And the next request to another. Also, I am not sure if I should expect this to be a round-robin or some other algorithm for load balancing.

I made traces with tcpdump from each of the containers and I am getting the following:
I can see the SYN packet to the docker_gw_bridge in TCP stream 0 and then TCP stream 1 from all of the containers
replica 1:

replica 2:

tcp stream 2:

replica 3:

Output of docker version:

docker version
Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:03:45 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:02:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
[root@localhost rmc]#

Output of docker info:

docker info
Client:
 Debug Mode: false

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 118
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: n01fmg6wf3rhaatkhfrm38gj5
  Is Manager: true
  ClusterID: ff75jzrb5qqxlablch78gpf49
  Managers: 1
  Nodes: 1
  Default Address Pool: 172.18.0.0/16
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.0.200
  Manager Addresses:
   192.168.0.200:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1127.19.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 1.795GiB
 Name: localhost.localdomain
 ID: GBCC:7IOA:74Y5:WKAO:KLKU:ZZKS:U2F4:NEIW:LKRA:IKPW:ETCP:S3Y6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  docker-quay.payten.com
  payten-docker.com
  127.0.0.0/8
 Live Restore Enabled: false

OS: CentOS 7.8
Kernel: 3.10.0-1127.19.1.el7.x86_64
Hypervisor: ESXi 6.5

[kolbma]

I can't find anything in the doc how the mesh does the load balancing, but I think it is one possibility to try to connect a bunch of endpoints and the first one wins.
If it would try one after the other there would be needed a lot of time and how long should it wait for trying the next.
It is not really known which replica does really work and will answer the SYN successful.
So maybe with 100 service replicas it would only send the SYN to 10... (just a guess)
Maybe there is also an internal state knowledge with success/fail counts and time.

Your replicas 2 and 3 are only tested for connectivity and the connection is closed immediately by the load balancer in the ingress mesh.
Have you tried it multiple times?!
You should see that there is some RR and the replicas doing the work should change.