Test Infrastructure

Tests rely heavily upon C++ inheritance and polymorphism to solve various requirements. The base class of every test is called Test, and attempts to place constraints and guidelines onto developers to support the following:

1. Force developers to code test logic in a uniform way.
2. Force developers to document tests, but alleviate the burden as much as reasonably possible.
3. Allow many ways for a test to report errors.
4. Bind and control resource lifetimes.
5. Check for hidden PCI or controller space register errors.

Force Coding in a Uniform Way

The base class to every test has a pure virtual method called Test::RunCoreTest(). This method must be implemented within all children otherwise they cannot be instantiated. RunCoreTest() is where each and every test contains its core test logic. This is the effective entry point into each and every test residing within the framework.

Force Documenting Tests

The children derived from base class Test are required to document the reason for the existence of the test, as well as other useful information. All tests are documented in a similar manner and will have a common output displayed when a user requests test details from the command line.

Specifically, the children must supply valid and relevant text by populating member variable Test::mTestDesc within the constructor of the child. This member variable is itself an object and requires initialization of the following items:

1. SetCompliance() - Which section and spec revision the test targets.
2. SetShort() – Provide a short 66 character description for test cases existence.
3. SetLong() – Provide an unlimited amount of text to thoroughly describe the test scenario.

This duty, albeit cumbersome but simple, will be vital to handing off test case descriptions to future generations.

Single Mechanism to Report Errors

There is one acceptable method of reporting test case errors and/or compliance violations to the framework. Due to the fact Test::RunCoreTest() returns a void, a developer must throw class tnvme/Exception/FrmwkEx() indicating an exception/error. The reason for this is to prevent a segmentation fault from occurring after a test failure has been indicated. The prevention lies within the constructor of class FrmwkEx. An exception is thrown from the lowest level of context, meaning test lifetimes will still be intact. The constructor's sole purpose is to dump valuable last minute debug data and to completely disable the DUT. The disabling of the DUT causes both dnvme and hdw to release all resources. If resources were not released at this point in time, a race condition would occur. The catching of exceptions are not within the lowest level of test lifetime context, which means those objects/resources have been freed back to the OS. However, hdw does not know this has occurred and may still think, for example, an IOCQ is still in existence and attempt to write a latent CE. But the write will be occurring into memory that may have been backed by a test lifetime memory resource, and thus a segmentation fault, or worse a core dump could occur. The throwing of the std::exception class or any other not derived from FrmwkEx will yield similar catastrophes; deriving from FrmwkEx will yield safe results and allow you to customize the error handling to suite your needs.

Check for Hidden Errors

During usual test case execution the test may create an environment and decide on the pass fail criteria of the logic. The framework adds further test case pass fail criteria by looking at the PCI register space and detecting any PCI error bits which have been set during the test run. The same holds true for the controller register STS. All registers which are checked after each test executes, are reset before the test runs. This wrapping guarantees if a bit has become set, it is the result of what has occurred within the test. For complete details see method Test::GetStatusRegErrors().