Test Infrastructure

Tests rely heavily upon C++ inheritance and polymorphism to solve various requirements. The base class of every test is called Test, and attempts to place constraints and guidelines onto developers to support the following:

1. Force developers to code test logic in uniform way.
2. Force developers to easily document tests.
3. Allow many ways for a test to report errors.
4. Bind resource lifetimes.
5. Check for hidden PCI or controller space register errors.

Force Coding in Uniform Way

The Test base class has a pure virtual method called RunCoreTest(). This method must be implemented within all children otherwise they cannot be instantiated. RunCoreTest() is where each and every test contains its core test logic. This is the entry point into each and every test residing within the framework.

Force Documenting Tests

The children derived from base class Test are required to document the reason for the existence of the test, as well as other useful information. All tests are documented in a similar manner and will have a common output displayed when a user requests test details from the command line.

Specifically, the children must supply valid and relevant text by populating member variable Test::mTestDesc within the constructor of the child. This member variable is itself an object and requires initialization of the following items:

1. SetCompliance() - Which section and spec revision is this test targeting.
2. SetShort() – Provide a short 66 character description for the test cases existence.
3. SetLong() – Provide an unlimited amount of text to thoroughly describe the test.

This duty, albeit cumbersome but simple, will be vital to handing off test case descriptions to future generations.

Allow Multiple Error Reporting Techniques

There are basically 2 acceptable methods of reporting test case errors and/or compliance violations to the framework. Because Test::RunCoreTest() returns a bool, a developer returns false upon errors and true when the test successfully passes. The other mechanisms is to throw a std::exception() from anywhere within the child class. All exceptions are caught by the base class and this is converted to the proper reporting mechanism back to the framework so that users see a test case ending in error when one throws.

Bind Resource Lifetimes

The concept of resource lifetime is complicated and the full details are covered in the link. Any resource created with test lifetime, which is dictated by a local function variable or a class level member variable will be destroyed/freed after the test completes. This is ensured, because the framework deletes each test object after it completes, regardless if it completed in success or failure. The deletion of an object destroys all contained resources if and only if proper guidelines are followed. This design allows, but highly shuns upon the use of malloc()/free(), and the new/delete operators. To make your code more likely to become accepted back into nvmecompliance it is recommended not to use these mechanisms. However the framework doesn’t safeguard nor prevent their use. If they must be used then simply follow safe programming practices by coding the corresponding free() or delete in the destructor or wherever appropriate. Be aware that this mechanism does not work well with lifetime promotions. In other words, if a variable is created with test lifetime using malloc() and then assigned to a resource with group lifetime, the lifetime promotion will not occur and a segmentation fault will most likely result. For resource allocation and lifetimes this design promotes the use of smart pointers. Each and every resource object contained in the framework will contained members and typedefs to aid in shared_ptr usage. The concept of a shared_ptr is that the pointer owns the resource. As long as there is a shared_ptr pointing to a resource, that resource is guaranteed to exist. Hence, declaring a local function variable or a class level member variable of type share_ptr will be cleaned correctly when the test completes, because each test instance is destroyed by the framework when it completes. This approach also works seamlessly with lifetime promotion. A resource with test lifetime will be correctly promoted to group lifetime if shared_ptr’s are utilized.

Important Resource Lesson

Class level member variables of any type are allowed, but it complicates the test logic. Each child class derived from base class Test must override the copy constructor and operator=() to prevent bitwise default methods which will be inserted by the compiler. Cloning objects are necessary to support automated resource cleanup between subsequent test case runs. This is the cleanup action of test lifetimes. Objects are cloned, then freed after each test completes to force resource cleanup. This cloning uses special copy construction and operator=() to achieve proper resource cleanup. The end result of a clone action should be to re-create a new object without doing any copying of any pointers. Do NOT do shallow or deep copies of any pointer. Any pointer is one of the following: share_ptr, weak_ptr, or C++ pointers using the new operator or malloc(). All newly cloned objects must achieve NULL pointers during construction or resource cleanup will not be possible. Thus, default copy constructors and operator=() cannot be used because they do bitwise copying. Thus all children must implement custom copy constructors and operator=() to disallow bitwise copying to achieve this requirement. In effect this allows proper resource cleanup by preventing the compiler from inserting copy constructors and operator=() methods.

Check for Hidden Errors

During test case execution the test creates an environment and decides on the pass fail criteria of the logic. The framework adds further test case pass fail criteria by looking at the PCI register space and detecting any PCI error bits which have been set during the test run. The same holds true for controller register STS. All registers which are checked after each test runs are reset before the test runs. This guarantees that if a bit has become set it is the result of what has occurred within the test. For complete details see method Test:: GetStatusRegErrors().