Framework

The purpose of the framework is to allow a developer to concentrate on test development and not have to worry about the overhead of managing hundreds of test cases. The framework is designed to provide basic processing for all tests. Some of these basic operations include:

1. Reporting test documentation to a user
2. Skipping undesired tests cases
3. Create a common error reporting technique
4. Create a common output logging naming convention
5. Allow the handling or ignoring of test errors

The purpose of dnvme is to open the door for user space applications to manipulate and control the NVMe API to a Device Under Test (DUT). However danger comes with this power. Generally dnvme does not check against all illegal conditions to which an application may place it. Therefore the dnvme API is a contract and if the contract is breached bad results may occur. You could see segmentation faults and most likely at some point be faced with kernel core dumps. But all is not lost if you follow the constraints of the provided framework.

The framework is built upon dnvme. It is there to open the kernel to user space manipulation, but also to place constraints upon a developer so that they won’t cause a core dump. When the NVMe specification makes the “shall not” statement, this usually means that if the stated action is performed, unknown and undesirable outcomes may result. This class of errors usually means that there is no way for the DUT to report an error and so causing this scenario will yield no insight into whether or not the DUT is compliant with the specification. Placing the test environment in one of these “shall not” states will not be productive, and so the framework tries to avoid such possibilities while still allowing a developer to control the hardware interface to its limit. One such scenario would be if a developer accidentally created discontiguous memory to back an ASQ. The specification states that all ASQ’s must be backed by contiguous memory. On the other hand, discontiguous IO queues are allowed. And so, the framework must allow the utilization of discontiguous memory but not for the purpose of backing ASQ’s. The important lesson to take away from this, is that if you find yourself deterring from the confines of the framework, you may be facing something you wish you had known about previously.

Multiple Threads

dnvme only supports single threaded interaction to each DUT. Mutex’s are utilized to guard against such misuse. The framework has been created to only utilize a single thread of execution. In compliance test suites this is seen as acceptable. However, this would not be acceptable in benchmark, throughput or performance type test applications, which tnvme is none of. Functionality, not performance, was a major design goal of this test suite, since it was seen that these two goals diverged at times during implementation.

Multiple Instances

dnvme only supports a single open call to each DUT. The framework also works to maintain this constraint. This implies that although multiple DUT’s cannot be the target of single tnvme instance, multiple instances of tnvme could be running concurrently to test multiple DUT’s on a single machine. The main constraint there is system resources. In reality, running multiple concurrent instances of tnvme won’t be plausible because during compliance testing the limits of a system will be strained to test the limits of the DUT. For example, if a test is running and needs to consume memory to fill the maximum allowed number of IO’s within a IO submission queue, 65536, how much memory will there be for the same test to run concurrently against another DUT within the same system?

Ignoring Failures

The default behavior of the framework is to execute the command line specified test case(s) until an error occurs. When an error is detected, tnvme exits and reports the issue. This behavior can be over ridden by command line option --ignore. If an error occurs however, and we are ignoring errors, then the next test within the same group cannot be executed. Rather the correct behavior, will be to execute the next test in the next group. The reason for this implicit skipping is to force the framework back to a well known state. The concept of groups allows tests to setup conditions for subsequent tests. This is powerful, but if a test fails whose sole purpose is to setup a scenario, then subsequent tests could never be expected to execute correctly. This could even cause a kernel core dump. The framework resets tnvme, dnvme and the DUT back to a well known state between each and every group. Skipping to the 1st test within the next group is sufficient to correct any partial test setup configurations.