Unable to promote replacement worker to manager

[kylegordon]

Running Docker 1.12.1, although have tried using 1.12.2 on the replacement too. Had to docker-machine rm swarm-master and add it to the remaining 3 node cluster as a worker.

└─(22:38:50)──> docker version ; docker info ──(Thu,Oct13)─┘
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 17:52:38 2016
OS/Arch: linux/amd64
Containers: 41
Running: 2
Paused: 0
Stopped: 39
Images: 5
Server Version: 1.12.1
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 112
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host null bridge overlay
Swarm: active
NodeID: 0wt3rh9l4hqsb76iv3muwu1ch
Is Manager: true
ClusterID: 5iv2laqoakm02gy2rju3r1drx
Managers: 4
Nodes: 4
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 10.2.2.61
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.17-boot2docker
Operating System: Boot2Docker 1.12.1 (TCL 7.2); HEAD : ef7d0b4 - Thu Aug 18 21:18:06 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.955 GiB
Name: swarm-node-01
ID: S4UC:ELN5:Q5UQ:ONYX:RPI4:QNKY:D7PQ:JZ2I:NKYN:CL5Q:BFOK:G62P
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 54
Goroutines: 177
System Time: 2016-10-13T21:42:37.775679335Z
EventsListeners: 1
Username: kylegordon
Registry: https://index.docker.io/v1/
Labels:
provider=vmwarevsphere
Insecure Registries:
127.0.0.0/8

This is the view from one of the regular manager nodes.

kyleg@wmlp1000:$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0wt3rh9l4hqsb76iv3muwu1ch * swarm-node-01 Ready Active Reachable
67dis5d6v8bitxyfnoom8039i swarm-node-03 Ready Active Reachable
98vm4nasl278ecmtz20rm4l77 swarm-master Ready Active
9m4rvgxche7f9xa2rske6mnzq swarm-node-02 Ready Active Leader
kyleg@wmlp1000:$
kyleg@wmlp1000:$ docker node promote swarm-master
Node swarm-master promoted to a manager in the swarm.
kyleg@wmlp1000:$
kyleg@wmlp1000:$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0wt3rh9l4hqsb76iv3muwu1ch * swarm-node-01 Ready Active Reachable
67dis5d6v8bitxyfnoom8039i swarm-node-03 Ready Active Reachable
98vm4nasl278ecmtz20rm4l77 swarm-master Ready Active Reachable
9m4rvgxche7f9xa2rske6mnzq swarm-node-02 Ready Active Leader
kyleg@wmlp1000:$

However, the view from swarm-master is different...

kyleg@kyleg-linux-desktop:$ eval $(docker-machine env swarm-master)
kyleg@kyleg-linux-desktop:$ docker node ls
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
kyleg@kyleg-linux-desktop:$ docker swarm join --token foo 10.2.2.61:2377
This node joined a swarm as a manager.
kyleg@kyleg-linux-desktop:$ docker node ls
Error response from daemon: rpc error: code = 2 desc = raft: no elected cluster leader
kyleg@kyleg-linux-desktop:~$

After waiting for a while, the problem persists. I've also tried using --advertise-addr and --list-addr to no avail.

The output of docker.log on swarm-master can be found at https://gist.github.com/kylegordon/0e144a115356e7a38092d48d90132601 where it repeatedly mentions the inability to resolve the address of member ID 38ab8bce6f20ccc0. As far as I can tell from docker node ls, there is no node with that member ID

Any clarity on this would be appreciated!

[aaronlehmann]

That's an interesting log. The node you added to the swarm received a snapshot, then after this it didn't know the address for another node it needed to communicate with. It asked a third node for the address, but that one didn't know either. Normally all managers know the addresses of all other managers. It's as if something was missing from this snapshot and it gave the managers amnesia.

Do you remember how you assembled this swarm? Were there any nodes that existed before that have since been removed or demoted?

[kylegordon]

If memory serves... it was assembled from an older swarm that was misbehaving. I can't remember the details of what went wrong with the old one, but it was early September (just after Docker Glasgow meetup!) and I recall using --force-new-cluster on one node, and then using docker swarm leave, and docker swarm join as a worker on the remaining nodes in order to join the new swarm. All four came online correctly, we deployed a service or two, and then promoted the remaining 3 to managers in order to handle traffic ingress across them all, as the hosts were spread over our vsphere nodes.

A couple of weeks further on, and for other reasons, I had to move the swarm nodes from one vsphere center to another. Each individual node was shut down, migrated, restarted, checked that it was active and reachable as a manager before doing the next one - in order to retain quorum. From a guest VM perspective, nothing changed there, but it did break my docker-machine configuration for managing these swarm members. It was at that point I ultimately ended up deleting a node and recreated it with docker-machine to gain docker-machine access to it, but now we have this funny state!

In answer to the last question - quite probably. Nodes have been demoted from the old swarm, and the problematic 'swarm-master' has been promoted, demoted and removed quite a few times now. docker-machine has been used to recreate it several times over too. The total number of nodes has never been above four though.

I'm happy to timebox this and just create a new cluster, if it proves to be a terminal edge case issue! Your call :-)

[aaronlehmann]

Thanks for the additional background. I think it's most likely that this is related to the use of --force-new-cluster, or one of the manipulations which happened after that. --force-new-cluster makes a node forget about its peers, so if there are still other nodes left over after --force-new-cluster, or their state is propagated somehow, I could see it causing a situation like this.

If you happen to find a sequence of steps that reproduces the problem, I'd be interested in hearing them, so we can find out if this is a bug in SwarmKit, or a bad state that we can discourage/prevent. But for now I think the best way forward is to create a new cluster.

[kylegordon]

I had a shot recently at recreating it, and unfortunately was unable to. The one thing that I did notice was that the node that was removed from the cluster and then had --force-new-cluster used on it still had all the details of the old nodes. Whilst it was indeed removed from the cluster, it hadn't forgotten about it's friends.

The end result of this was that the new cluster had 4 healthy nodes and the 4 'dead' nodes from the previous cluster. They were all removed fine though, and the cluster operated as expected.