IP for one service's task is in rotation for another service's VIP

[evanp]

Description

I have two services, and one of them has one of the other's task's IP in rotation for its VIP.

Steps to reproduce the issue:

1. Create three services in a swarm, including one with a shell.
2. In the shell, use dig tasks.service to get the IP addresses of the tasks of the different services.
3. Compare the results

Describe the results you received:

One service had 3 replicas, but 4 IPs listed. The other service had 2 replicas and 2 IPs listed. One of the IPs in the 2-task service was tacked on to the 3-task service's IPs.

Describe the results you expected:

Each service would have a distinct list of IPs.

Additional information you deem important (e.g. issue happens only occasionally):

Testing the output of the overlapping IP shows that it actually is a task for the 2-task service.

Output of docker version:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 7
 Running: 5
 Paused: 0
 Stopped: 2
Images: 5
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 49
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge host overlay
Swarm: active
 NodeID: edcllto3t6fof8k9r9c0uxcl5
 Is Manager: true
 ClusterID: dfeiojl0vyi6c3x8j38zwiyqx
 Managers: 3
 Nodes: 32
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.134.18.127
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-38-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953 GiB
Name: prod14
ID: CVXT:L6PJ:5RYA:CGQE:VTGA:MMQU:4NMN:ANVS:P6RQ:ED4I:5LPX:3I5B
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: fuzzyioproduction
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 provider=digitalocean
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

root@fd7e0946c64f:/usr/src/app# dig tasks.api                                               

; <<>> DiG 9.9.5-9+deb8u6-Debian <<>> tasks.api
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 389
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tasks.api.         IN  A

;; ANSWER SECTION:
tasks.api.      600 IN  A   10.99.86.31
tasks.api.      600 IN  A   10.99.86.78
tasks.api.      600 IN  A   10.99.86.90
tasks.api.      600 IN  A   10.99.86.91

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Tue Sep 20 04:48:30 UTC 2016
;; MSG SIZE  rcvd: 127

root@fd7e0946c64f:/usr/src/app# dig tasks.stats

; <<>> DiG 9.9.5-9+deb8u6-Debian <<>> tasks.stats
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51264
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tasks.stats.           IN  A

;; ANSWER SECTION:
tasks.stats.        600 IN  A   10.99.86.25
tasks.stats.        600 IN  A   10.99.86.78

;; Query time: 1 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Tue Sep 20 04:48:35 UTC 2016
;; MSG SIZE  rcvd: 83

[evanp]

It seems like the IP for task that was stopped is still mapped to that task. I've seen this in other cases, such as when using docker service rm or docker service update. I hope I can figure out a solution here.

[evanp]

So, I think the big issue here is docker service update. I left it out of this issue, but there's more detail in #26772. I'm going to close this one in reference to that one.