Proposed RFC Feature editor_test.py reconciles included Report.results against actual Report summary

[smurly]

Summary:

Currently when a test runs using editor_test.py (batch/parallel), the Editor Python Binding (EPB, hydra) script contains some number of Report.result and Report.critical_result lines. If the script ends early without having called those lines there is no accounting that tests were skipped or missed. Only if a log is successfully captured can we see which Report lines actually got logged.

What is the relevance of this feature?

In the event that the application under test (AUT) exits early but does not indicate error or is non-responsive and fails to log information before being terminated for timeout, we have no way of assessing test lines that were skipped or missed. We can only see that the overall test completed with status timeout or ended with exit code (expected or otherwise). If the test exits without indicating an explicit failing result, the overall assessment would then be passing even if some number of test lines are skipped.

Feature design description:

• before executing the AUT we should parse the EPB script for expected results so that on completion we can reconcile expected to the actual Report summary. This would prevent us from failing to notice skipped test functionality.

Technical design description:

• Parse the EPB script for a list of Report.result lines to create an accounting
  • code branching will add complexity for e.g. if-statements
  • asserts are also important to verify whether they were evaluated (may require spying, or profiling, or using a Report.Assert?)
• After execution of the AUT with the EPB script, compare the result summary to the expected and reconcile the difference
• Indicate discrepancies in tests reporting with a mechanism that alerts users to missed testing

What are the advantages of the feature?

• Having this may prevent false perception about the completeness of testing; we will know if tests lines are not running.
• Provides a consistency check that the full set of assertions in a script ran, and did not exit early.

What are the disadvantages of the feature?

• Some tests may intentionally be within branching code that either intentionally doesn't run for various conditions or is meant to run multiple times for a list of values. This would make reconciliation difficult to achieve. (implementation complexity)
• It may be more expensive in resources to build this than to use mechanisms to manually ensure testing is completed as expected. (auditing test health, unlikely to be done except when the test or feature fails for other reasons)

How will this be implemented or integrated into the O3DE environment?

Work would be done within the editor_test.py batch/parallel framework and should be transparent to test implementers.

Are there any alternatives to this feature?

• manual audit of test results
• A structured expectations library may simplify this: EditorTest expectations library o3de#8874

How will users learn this feature?

• possibly docstrings in editor_test.py or report summary output differences

Are there any open questions?

[Kadino]

This can help add warnings for when asserts are not being reached, or could never be reached.