Brittle unit tester testing is still brittle

[tngreene]

How the unit tester decides if there was an error has gone through many iterations - each hoping to not produce false negatives or positives.

The latest iteration (7a96f1d)

``python
#First line of output is unittest's sequece of dots, E's and F's
if 'E' in out_lines[0] or 'F' in out_lines[0] or num_errors != 0:

Hoped to use the first line's series of .'s and E's would finally solve the problem, but it turns out that that is brittle too!

Sometimes an exception can be thrown making the first line be "try".

The order of the output of the test has always seemed incredibly strange and out of order, but surely there is a pattern we can count on.

Another solution may be to see if we can get the actual test result class back instead of just the output and we can just find a test results errors with its ``errors`` member. https://docs.python.org/3/library/unittest.html?highlight=testresult#unittest.TestResult

[tngreene]

Good news! The answer probably lies in
tests/init.py's runTestCases function

    for testCase in testCases:    
        suite = unittest.defaultTestLoader.loadTestsFromTestCase(testCase)
        expected_logger_errors += testCase.expected_logger_errors
        expected_logger_warnings += testCase.expected_logger_warnings
    
    unittest.TextTestRunner().run(suite)

We can customize our TestRunner to produce less garbage, and use the return value of run to directly get the information we want out of it.

In addition, we may be able to use the built in expectedFailures and unexpectedSuccesses instead of our own hack that was half used and now currently abandoned.

Even without the ability to completely control the output, with a captured TestResult we'll be able to come up with our own smartly written-stupid string to compare to!

Alternatively, we look up a way to share some chunk of memory (virtual, real, or disk) to pass the information between the processes and there won't be any more string matching ever!

[tngreene]

Ideas for the much more fun - non-string parsing way

• Multiprocessing sharing a large read only object between processes
• Sharing a complex object between python processes

My guess is that we'll go with more string crunching until it fails again and breakdown and do the fun things.

[tngreene]

Another problem: Currently we say "Assert there was 1 error", but it could be any error. As I've found in fixing translation_bone_rules and rotation_bone_rules false positives can arise:
If error B is what you're expecting,

• Error A can occur first, hiding whether error B occured at all or not or would have occured at all.
• Error B can not occur, but error C does occur causing "num_errors == 1" to be true

We could really use persistent error codes like we have in WED.

[tngreene]

Also! The logger doesn't get cleared automatically between test cases. This was discovered around the time of fa80228 with rotation_bone_rules.test.py

test_5 was expecting to test against a fixture. When it failed, test_6 also failed, despite passing normally because the logger still had left over errors. See also #303

[tngreene]

In addition! It would be really nice to be able to filter the output a bit to remove Blender's miscellaneous output of irrelevant debug messages. The addition of a -v[vv] or --verbose [level] to control this would be excellent. The ability to control the output entirely could be a potential solution without the above descriptions!

These seem relevant to the problem

• Discard stdout / stderr of program under test, but keep unittest output
• python - how can I redirect the output of unittest? Obvious solution doesn't work

See #408

[tngreene]

Another burn today: Good news, there are two new ways to strength the unit tester without much work

1. Remove allowing the LOGGER string to be None.
2. Use the test runner's result to get the errors and failures not picked up by our XPlaneLogger.
3. It would also be a good idea to remove our (brief) mentions of "expected_errors" because test cases already have that and we shouldn't re-invent wheels.

[tngreene]

Also! We could replace the logger with logger! That would be fun and create a lot better filtering for debug during unit testing. Perhaps it could even help with printing the outcomes of the tests as well.

[tngreene]

Closing and splitting the other bits into smaller bugs!