Manually trigger an election (exploratory, unfinshed feature in development)

[christophermaier]

To help work around some issues in elections, as well as to help
investigate them further, this commit introduces the beginnings of a
potential feature for allowing operators to manually trigger a service
election.

The current incarnation relies on the HAB_FEAT_TRIGGER_ELECTION
feature flag. A Supervisor operating with this flag enabled will
initiate a new election for a service group if a special sentinel file
for that group is found on disk.

To trigger a new election for the service group foo.default, for
example, create the file

/hab/sup/default/data/trigger_foo.default_election

(That is, a file with the name trigger_${SERVICE_GROUP_NAME}_election.)

This should be done on a Supervisor that is actually running the
foo.default service group.

As the Supervisor's main loop executes, it will respond to the
presence of such a file by initiating a new election. The sentinel
file is also deleted, to prevent endless cycles of elections. If the
file is created in such a way that the Supervisor cannot delete it, no
election is triggered.

To monitor this feature in logs, add
habitat_butterfly::server::election_trigger=trace to your RUST_LOG
environment variable (or use info for less noisy log output).

cc: @irvingpop @dmccown

[chef-expeditor]

Hello christophermaier! Thanks for the pull request!

Here is what will happen next:

1. Your PR will be reviewed by the maintainers.
2. If everything looks good, one of them will approve it, and your PR will be merged.

Thank you for contributing!

[raskchanky]

One small comment regarding the nesting level but otherwise looks fine to me

[mwrock]

This looks good to me. I am guessing that if this feature gets baked, we would make the election trigger via the sup cli rather than the existence of a file?

[raskchanky]

@mwrock I think that was the end goal, yeah, with proper channels and such for communicating between pieces. The file thing was just something easy to see if this is actually useful.

[christophermaier]

@mwrock @raskchanky Yes, that's exactly the plan.