terminating a one of three etcd masters caused massive fail

[crewjam]

The chain of events:

1. Terminate a master node terminating a one of three etcd masters caused massive fail #1
2. Notice that etcd on master node fleet failure should force a reboot or shutdown or something #2 hangs with:

$ fleetctl list-units
2014/10/11 02:38:21 ERROR fleetctl.go:171: error attempting to check latest fleet version in Registry: timeout reached
2014/10/11 02:38:21 INFO client.go:278: Failed getting response from http://127.0.0.1:4001/: cancelled
Error retrieving list of units from repository: timeout reached

1. Notice that etcd on master node #3 fails with:

$ ssh 10.10.126.61 etcdctl ls /
Error:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

1. Issue systemctl restart etcd on master fleet failure should force a reboot or shutdown or something #2
2. Notice that fleet is all screwed up:

$ fleetctl list-units
UNIT                    MACHINE             ACTIVE      SUB
gerrit-app-presence@1.service       268251a3.../10.10.126.61    inactive    dead
gerrit-app-presence@2.service       4e283163.../10.10.146.25    inactive    dead
gerrit-app-presence@3.service       836de616.../10.10.1.191     inactive    dead
gerrit-app@1.service            268251a3.../10.10.126.61    inactive    dead
gerrit-app@2.service            4e283163.../10.10.146.25    inactive    dead
gerrit-app@3.service            836de616.../10.10.1.191     activating  start-pre
gerrit-data-volume-git@1.service    268251a3.../10.10.126.61    activating  start-pre
gerrit-data-volume-git@2.service    4e283163.../10.10.146.25    activating  start-pre
gerrit-data-volume-git@3.service    836de616.../10.10.1.191     active      running
gerrit-db-data-volume@1.service     836de616.../10.10.1.191     active      running
gerrit-db-data-volume@2.service     4e283163.../10.10.146.25    activating  start-pre
gerrit-db-data-volume@3.service     268251a3.../10.10.126.61    activating  start-pre
gerrit-db-gerrit-amb@1.service      268251a3.../10.10.126.61    activating  start-pre
gerrit-db-gerrit-amb@2.service      4e283163.../10.10.146.25    activating  start-pre
gerrit-db-gerrit-amb@3.service      836de616.../10.10.1.191     active      running
gerrit-db-pod@1.service         836de616.../10.10.1.191     active      running
gerrit-db-pod@2.service         4e283163.../10.10.146.25    active      running
gerrit-db-pod@3.service         268251a3.../10.10.126.61    active      running
gerrit-db@1.service         836de616.../10.10.1.191     activating  start-pre
gerrit-db@2.service         4e283163.../10.10.146.25    inactive    dead
gerrit-db@3.service         268251a3.../10.10.126.61    failed      failed
gerrit-pod@1.service            268251a3.../10.10.126.61    active      running
gerrit-pod@2.service            4e283163.../10.10.146.25    active      running
gerrit-pod@3.service            836de616.../10.10.1.191     active      running
nginx-presence-dns.service      4e283163.../10.10.146.25    activating  start-pre
nginx-presence@1.service        268251a3.../10.10.126.61    inactive    dead
nginx-presence@2.service        4e283163.../10.10.146.25    inactive    dead
nginx-presence@3.service        4e283163.../10.10.146.25    inactive    dead
nginx@1.service             268251a3.../10.10.126.61    activating  start-pre
nginx@2.service             4e283163.../10.10.146.25    activating  start-pre
nginx@3.service             4e283163.../10.10.146.25    activating  start-pre