Add -u|--unanimous flag: easily identify whether all engines agree on output

[dilijev]

To help with automatic comparative testing between engines, add a -q|--quorum flag. Behavior will be as follows:

• If all engines under test produce identical output (logic already exists to test this in the collate option), print no output and set exit code to 0.
• If any engine disagrees, show output, and set exit code to 1
  • Alternatively: set the exit code to (1 + the index of the engine in the config file) to more easily identify which engine disagreed. Perhaps -1 if more than one engine disagrees or there is a greater than 2-way split on the outputs.

/cc @bterlson

[bterlson]

I think communicating the hosts with different results via exit code isn't a great design. Probably the stdout should contain this info in an easily parseable format?

[dilijev]

@bterlson Yeah, communication via the exit code was a bit of a stretch. I'd say the output format is not necessarily super important for a first pass at this feature, but an easily-parseable format would definitely be good.

I'm trying to decide whether the current default collated output is not, in fact, easily parseable. I suppose the only problematic things are the colors and if your test output has a line that starts with #### you would be in trouble. Maybe strip the colors and indent the entire output block so only the engine identification lines are left-justified?

[dilijev]

Started work on this here: https://github.com/dilijev/eshost-cli/tree/quorum

[dilijev]

Renaming option to --unanimous as per feedback in #35