-
Notifications
You must be signed in to change notification settings - Fork 99
Runnable Test
A runnable test is a concept that allows a developer, on a per test basis, to indicate to the framework whether a test can run based upon any criteria. Every developed test case indicates to the framework an answer to the question: Can the test execute now? While answering the question, the framework forces constraints upon the answer as ordered from cmd line arguments. The final answer to the question must not violate the these constraints and must be provided in a child's overriding implementation of method Test::RunnableCoreTest().
Currently there is only 1 constraint possible and it is handed down to every test via the cmd line option --preserve.
Every child must override RunnableCoreTest() base class method, failure to do so results in a test failure. This is the place each test case must answer the question outlined above. There are 3 possible outcomes to this question and are detailed as follows.
The most simple outcome to the answer forces the framework to immediately call RunCoreTest() to execute the test. However, if this test is permanently destructive and cmd line option --preserve disallows this, then TRUE cannot be returned as the answer.
There can be many reasons for this outcome, but this is not considered an error. Rather the framework skips the test case as if it had been supplied in the --skiptest cmd line filename.
If the test is permanently destructive and cmd line option --preserve disallows this, then FALSE should be the answer to the question. Anything permanently destructive forces the DUT to be permanently changed through a cold-hard reset. For example, writing to media, formatting the media, downloading firmware are a few examples.
Other reasons for returning FALSE could be a dynamic runtime determination that there aren't enough resources to perform the test case. However, if this is the developers reason, perhaps the design of the test needs to be rethought to work for any and all possible devices which can be manufactured.
This answer causes the framework to fail the testing procedure as long as the cmd line has not supplied option --ignore. In this case, perhaps as the result of extracting necessary data from the device to make this decision, the DUT incurred an operational failure.
There are no restrictions to the logic embedded within RunnableCoreTest() except those which are also imposed by RunCoreTest(). In fact the same code called within RunCoreTest() can also be called from RunnableCoreTest().