Provide More Information About Runner Results In Status Endpoints

[AntsInMyEy3sJohnson]

Context

Hazeltest's /status endpoint provides information on the internal status of each runner, such as the number of maps (or queues), the number of runs, and whether the runner has finished.

Problem

• Apart from whether the runner is done, the information available on the status endpoint is a subset of each runner's configuration, hence querying it doesn't add a lot of value beyond knowing whether the runners are done
• More specifically, the information the status endpoint currently provides is not sufficiently exhaustive and detailed to have reporting built on top of it

Solution

• Make the runner's test loops gather much more information:
  • Internal status (e.g. How many runs out of x has this runner done?)
  • Results of operations on Hazelcast (e.g. In this queue runner, how many of x put attempts made thus far have been successful, and how many have been rejected because the target queue was already full?)
• Offer information on status endpoint

Additional Remarks

• The kind of aforementioned information on the runner's test loop's results is the foundation for reporting logic -- the same result might translate to test success in one team, but might constitute test failure in another, depending on the configuration of their Hazelcast clusters --> It would be futile to attempt to provide this reporting logic in Hazeltest itself
• To record every step each runner makes will likely impact application performance and increase resource usage, both in terms of CPU and memory --> Providing the full set of information in cases where it is not required would be inefficient
• Hence, it may be useful to provide different "granularity settings" for the status gathering, for example, basic, intermediate, and full mode
  • basic mode could include only the information available today, in which the only dynamic piece of information is whether the runner has finished --> sufficient to build simple reporting (All tests done?) and has no impact on application performance and resource usage
  • On the other extreme side, full mode could gather information on every single action each test loop of each runner performs, and record whether this action was successful --> necessary if one were to implement very sophisticated reporting, but would come at the cost of slower test loop execution and a heavier resource footprint
• In this way, a user can decide how much information is required, and configure status gathering functionality accordingly; it might even be useful to allow status gathering to be disabled altogether
• The foundation for more flexible status gathering has been laid in Improve Test Loop Status Data Gathering

[AntsInMyEy3sJohnson]

First Chaos Monkey now available. Add more information about monkey, too?

[AntsInMyEy3sJohnson]

Test loop status for both batch and boundary test loops should be exposed, too.