Added hooks in interface layer to facilitate a rolling restart.

[petevg]

Allows Zookeeper to perform an automagic rolling restart, rather than
requiring an ops person to do the restart manually.

@juju-solutions/bigdata

[ktsakalozos]

What would happen if the leader changes?

[petevg]

Docs on Zookeeper leader election are here: http://zookeeper.apache.org/doc/current/recipes.html#sc_leaderElection

Per convo in Hangout w/ @ktsakalozos, we do not currently handle the case where a Zookeeper leader election happens without triggering an event that Juju is wired to notice (e.g. the process crashes, but the machine does not go away). If that happens, Juju may have the wrong node flagged as the Zookeeper leader.

Unfortunately, there really isn't an obvious way to handle that. Juju reactive handlers trigger when a Juju event happens; by definition, they can't trigger when something happens that Juju doesn't register as an event.

We've decided that the best thing to do is to document this possibility, and open a ticket to deal with it if it turns out to be something that happens a significant number of times in the field. (An operator could fix things in the meantime by doing a manual rolling restart on the cluster).

[johnsca]

Well, we could have the unit respond to ZK events (leader change) and use juju-run on the unit to invoke one of the hook handlers. It's certainly something to keep in mind, but as you said, we can defer it to see how big of an issue it is in practice.

[kwmonroe]

Looks like comments have been address, and group think says to watch this space for edge-case leadership changes.

Sounds good to me.