Test Numbering

Test numbering within one group will never conflict with the numbering of another because tests case reference numbers must be fully qualified by specifying the group from which it belongs. So test case 0.0.0 within group 4 and within 5 will be fully qualified as 4:0.0.0 and 5:0.0.0. Take note of the GRP_NUM: preceding the test reference. This format uniquely identifies every test case within tnvme and allows for future growth and insertion.

One can build highly functional test scenarios by leveraging an understanding of prior tests. The idea is that prior tests are the configuration or setup of a test scenario. This power also has a downside in that we can no longer just choose to run any given test within any group and expect it to pass. We would have to run the tests which setup the environment for those tests which have prerequisites to succeed. The framework understands these inherent prerequisites so as long as some simple guidelines are followed. Building or architecting tests can be done in any of 3 ways.

Zero Dependency

A zero dependency test is one which can be run singularly and it can be expected to pass. It has zero setup dependency requirements. Tests of this scheme are numbered GRP_NUM:x.0.0, where x={0...(n-1)}. These tests reside at what is called xLevel.

Configuration Dependency

A configuration dependency test is one which can only be expected to pass if and only if the configuration test was run previously, not immediately prior, but sometime prior. It only has to run at least once previously to guarantee resources have been created which are assumed to be present by subsequent tests, i.e. other tests at this level. Tests of this scheme are numbered GRP_NUM:x.y.0, where x={0...(n-1)}, and y={0...(n-1)}. These tests reside at what is called yLevel. The configuration test must be the first test within yLevel, i.e. GRP_NUM:x.0.0 must be the configuration test case.

The framework automatically handles the dependencies at yLevel on your behalf. If say for example test GRP_NUM:2.4.0 fails and the test is desired to be executed for debugging purposes, then one only needs to specify the failing test on tnvme's cmd line. The framework will automatically handle running test GRP_NUM:2.0.0 first to force the configuration to exist prior. Thus typing "tnvme --test=GRP_NUM:2.4.0" will cause 2 tests to execute in this order: GRP_NUM:2.0.0, GRP_NUM:2.4.0.

Sequence Dependency

A sequence dependency test is one which can only be expected to pass if and only if all of the prior test cases have run in sequence at this level. Tests of this scheme are numbered GRP_NUM:x.y.z, where x={0...(n-1)}, y={0...(n-1)}, and z={0...(n-1)}. These tests reside at what is called zLevel. Each and every prior test at this level adds to the resources which are assumed to be present for all subsequent tests which follow.

The framework automatically handles the dependencies at zLevel on your behalf. If say for example test GRP_NUM:2.4.3 fails and the test is desired to be executed for debugging purposes, then one only needs to specify the failing test on tnvme's cmd line. The framework will automatically handle running the appropriate sequence of tests prior to the test of interest. Thus typing "tnvme --test=GRP_NUM:2.4.3" will cause 5 tests to execute in this order: GRP_NUM:2.0.0, GRP_NUM:2.4.0, GRP_NUM:2.4.1, GRP_NUM:2.4.2, and GRP_NUM:2.4.3. Please note that test GRP_NUM:2.0.0 was executed according to the configuration dependency at the yLevel.

If the sequence dependency is desired, but the configuration dependency is not, then numbering a sequence of tests as x.0.z will produce the desired results. For example, if test case GRP_NUM:2.0.3 fails and the test is desired to be executed for debugging purposes, then one would specify "tnvme --test=GRP_NUM:2.0.3". The framework will automatically execute 4 tests in order: GRP_NUM:2.0.0, GRP_NUM:2.0.1, GRP_NUM:2.0.2, GRP_NUM:2.0.3. Please note in this case the configuration dependency is avoided.