integration: Be choosy about which manager to demote

In rare cases, TestDemoteDownedManager can fail. The scenario involves
choosing the third manager to demote, and stopping it before it has a
chance to write any entries to its WAL. When it starts back up, it
doesn't know of any other members in the cluster (and no other members
will connect to it since it has been removed), so it has no way of
knowing it has been removed.

To avoid this case, we can simply avoid choosing the third manager.
Adding that manager is a synchronization point which guarantees that the
first and second managers will be caught up, and removing either at this
point wouldn't pose the same problem. In practice, we end up removing
the second manager, since the first is the leader.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

integration/integration_test.go

@@ -343,10 +343,13 @@ func TestDemoteDownedManager(t *testing.T) {
 	leader, err := cl.Leader()
 	require.NoError(t, err)
 
-	// add a new manager so we have 3, then find one (not the leader) to demote
+	// Find a manager (not the leader) to demote. It must not be the third
+	// manager we added, because there may not have been enough time for
+	// that one to write anything to its WAL.
 	var demotee *testNode
 	for _, n := range cl.nodes {
-		if n.IsManager() && n.node.NodeID() != leader.node.NodeID() {
+		nodeID := n.node.NodeID()
+		if n.IsManager() && nodeID != leader.node.NodeID() && cl.nodesOrder[nodeID] != 3 {
 			demotee = n
 			break
 		}