Docker Swarm should allow demoting down to a single master.

[kelseyhightower]

Output of docker version:

Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd-unsupported
 Built:        Fri Jun 17 21:21:56 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd-unsupported
 Built:        Fri Jun 17 21:21:56 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.12.0-rc2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 13
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: active
 NodeID: 5y4a52x18pg39n81ce97mdmf6
 IsManager: Yes
 Managers: 2
 Nodes: 3
 CACertHash: sha256:7f84cc99b3cf8c6f7feeb4327e985a3a587db7dd36b6326bcdf69383cf5d929e
Runtimes: default
Default Runtime: default
Security Options: apparmor seccomp
Kernel Version: 4.4.0-22-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.305 GiB
Name: ss-1
ID: 3P3V:JNEQ:PKSQ:75UF:VJCB:U2CD:NCMV:HTYN:I62I:7AT2:BFHM:LP6E
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

I'm running a three node cluster on GCE

Steps to reproduce the issue:

1. Initialize a new swarm cluster
2. Run docker swarm join IP:PORT --manager on two addition nodes
3. Demote 1 node at a time with docker node demote

Describe the results you received:

# docker node ls
ID                           NAME  MEMBERSHIP  STATUS  AVAILABILITY  MANAGER STATUS
0m9ptjt4qceaadsqfv0bsvkyl    ss-2  Accepted    Ready   Active        Reachable
31seal0wmf44l3q0x9nk9ddww    ss-3  Accepted    Ready   Active        Reachable
5y4a52x18pg39n81ce97mdmf6 *  ss-1  Accepted    Ready   Active        Leader

docker node demote 0m9ptjt4qceaadsqfv0bsvkyl
Manager 0m9ptjt4qceaadsqfv0bsvkyl demoted in the swarm.

docker node demote 31seal0wmf44l3q0x9nk9ddww
Error response from daemon: rpc error: code = 2 desc = raft: member cannot be removed, because removing it may result in loss of quorum

Describe the results you expected:

docker node demote 31seal0wmf44l3q0x9nk9ddww
Manager 31seal0wmf44l3q0x9nk9ddww demoted in the swarm. Warning you are running a Swarm cluster with a single node, <insert why this is bad> 

Additional information you deem important (e.g. issue happens only occasionally):

[kelseyhightower]

The reason for this issue is to provide a consistent experience to Swarm users. If you can start with a single master and scale up, then it should be possible to scale back down to a single node since these commands should be managing the raft membership as well.

[abronan]

Agreed that this would provide with a much better experience even though I think this scenario should not be recommended.

From my understanding (but I could be wrong), this is a limitation of the raft library, as when you remove a member, this only sends a ConfChange message to remaining members. In this case removing the last member would kill the quorum, thus the operation is not allowed (and the node to be removed does not receive ConfChange messages anyways so the first node is waiting for an answer from a ghost).

Potential solutions could be:

• Use the same process involved by --force-new-cluster and put the second node in the removal list while manipulating the logs to force a ConfChange on the first node. impact: nothing to change, this can be done with the library
• Make the configuration change workflow symmetrical and allow the node to be removed to receive a ConfChange and answer with a special type of message (while ignoring log append proposals). impact: this would involve changing the library and the internals

That is something that needs to be discussed with the etcd team as this can benefit anyone using the library.

[kelseyhightower]

@abronan Your suggestions are spot on and I'm sure the etcd can help work things out.

[aaronlehmann]

Note that this was fixed in the upstream swarmkit repository: moby/swarmkit#1091

The next time swarmkit is vendored, this issue can be closed.

[abronan]

/cc @tiborvass @thaJeztah

[tiborvass]

@abronan this will stay open until swarmkit is vendored.

[abronan]

#23312 includes the fix to scale down to one manager. Closing.