Thinking about combining tests

[bkatiemills]

Now that we are effectively examining single tests, one of the big problems left to solve is how to study the combination of groups of tests by ANDing and ORing their results. In this PR, I draft some functions to help us explore every possible AND/OR combination of a collection of tests.

combineTests in combineTests.py is the top-level function for doing this. It takes as an argument a list of lists, where the first index counts tests and the second counts profiles, since this is exactly what we are currently generating around line 206 in AutoQC.py. It uses the other functions in combineTests.py to enumerate every possible way to combine every possible subset of tests, and returns a list of lists that describe each combination and the resulting assessment of each profile.

There is a dummy execution of this function in combineTests.py; running that file will hopefully illustrate what I'm doing.

All these new functions need to be tested before this PR is complete and ready for merge.

How to read test partitioning syntax

Tests are collected in groups to AND together, and groups to OR together, and these groupings are expressed in lists. For example, a partition expressed as [[0,3], [1], [7]] means (0 AND 3) OR (1) OR (7). Note not all tests need be used in a given partition.

[bkatiemills]

Wrote some tests - but it occurs to me that not all permutations are currently getting generated.

[bkatiemills]

It also occurs to me that there's no current way to express: A and (B or C). But, I think this can be easily fixed by taking two passes at interpreting the structures we're creating: once the way described above, where

[A,[B,C]] = A or (B and C)

and then a second pass swapping the logic:

[A,[B,C]] = A and (B or C)

[bkatiemills]

Addressed permutations issue in 5db74e2, but need to tidy up & write more tests, also think about and/or interpretation as commented above; sometimes necessary, but not always - consider [[A,B]] vs [[A],[B]]; swapping ands and ors would double count in these symmetric cases.

[bkatiemills]

Alright - that ought to do it for a first pass. ToDo:

• write some more tests (such as Code up Argo spike test #25 Code up Argo gradient test #26 Code up WOD gradient check #27 Code up EN spike and step check #28) so we can see this stuff in action
• re-think log reporting to accommodate test combinatorics
• consider how to stop computing time from blowing up rapidly as the number of tests grow.