Evaluate what conditions should automatically trigger a service election

[jsirex]

This issue is for discuss Chef Habitats' election best behavior .

I see that stop/start service doesn't trigger election.
Now elections only triggered when leader supervisor goes down.

Probably, we should trigger a new election:

1. Every time service topology changes (for example, if service stopped, configuration changed, but supervisor is still running)
2. On health-check failed (not sure we should, but looks interesting)
3. Periodically (not sure we should, but looks interesting). Example: pool of workers with changing leader on the fly depending on current member capacity)

My real use-case example now: I'm building postgresql streaming replication service. And I need a master node always be master. So if it goes down - new temporary leader will be elected (postgres stays in readonly mode). When master goes back online - it should be the leader.
To achieve this I create suitability hook which always returns 10 for a leader, 5 for a follower, and 0 when uninitialized.

Picture gets more complicated when you starting think about "should suitability hook be triggered before service start, after start, after health-check passed, after post-run?

[jsirex]

I'm ended up with decision that nowadays it is impossible to use leader-follower topology at all.
Here is classic and simple use-case:
I have 3 postgresql servers running in leader topology. I set them so there is primary and 2 hot stand by with synchronous replication.

Now I want to upgrade services to something minor. But stopping the service is not triggering election, so it not possible to upgrade without downtime. I also cannot stop supervisor itself, because it is running other services.

Digging into source code I found a "hack" with HAB_FEAT_TRIGGER_ELECTION enabled to manually trigger it. Assume this flag is for debugging/developing and should not be considered as best practice.

[davidMcneil]

Relates to #5325. I think it would be useful to have a command to manually trigger a service election (#7787). I also think it would be useful to evaluate under what conditions we should automatically trigger an election. I am going to change the title of this issue to reflect that broader scope.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[mwrock]

not stale

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.