-
Notifications
You must be signed in to change notification settings - Fork 99
Resource Lifetime
In the most general sense, resources are those objects tnvme uses to setup and test a DUT. Resources are commands from any command set, any type of queue, and memory (both contiguous and discontiguous with various alignment requirements). The quickest way to determine if a C++ class is considered a resource is to investigate if it has a base class of Trackable. Any class derived from the Trackable class is therefore "trackable" by the resource manager. But this isn't the entire story, not all classes are meant to be instantiated, rather some class definitions might inherit from Trackable but aren't meant to be used directly. These intermediate classes are present to provide building blocks to give extra characteristics to child classes. A good example of an intermediate class is the NVMCmd class which is the parent of all commands within the NVM command set; another is the AdminCmd for the administrator command set. To determine if a class is both meant to be instantiated directly and is also considered a resource you need to investigate whether or not the resource manager allows the creation/allocation of it.
The resource manager is called gRsrcMngr and is of type class RsrcMngr. It is a global object, which is why it has a ‘g’ prefixed. It is created automatically by the framework and is guaranteed to exist before any test assumes control. It is also a singleton, meaning only 1 object of its kind will ever exist for each running instance of tnvme. The main job of the gRsrcMngr is to allocate resources and allow their deallocation when their lifetime has expired. All objects born of the gRsrcMngr have what is called group lifetime. Those resources are guaranteed to exist/live for the entire duration of a group's processing. In other words, after all tests have executed within a group, then and only then, will the framework force the deallocation of all resources born of the gRsrcMngr. The framework simply notifies the gRsrcMngr to initiate the deallocation of every resource.
Test objects represent a test cases scenario. A test is given control when the framework calls its RunCoreTest () method. Any test within a group can call upon the gRsrcMngr to create resources for their benefit. A test simply calls method gRsrcMngr->AllocObj () and supplies the type of object/resource for which it desires. Those resources will now live until all tests within the parent group have either completed execution successfully, or an error occurs, which causes control to exit the group prematurely. The group will exit even if the command line specifies to ignore errors. Please recall --ignore causes the next test within the next group to execute when an error is detected.
Group lifetime is powerful. Groups can be arranged such that tests exist for the sole purpose of setting up the test infrastructure on behalf of higher numbered tests within any given group. Higher numbered tests may specify prerequisites that resources must already have been created and ready to use in order for them to be effective and pass the criteria. Lower numbered tests may interact with the gRsrcMngr, when doing so they are forced to assign identifiers to the objects/resources which are being created. They can also proceed to setup and initialize those resources. Subsequent, higher numbered, tests can include the header file of those lower numbered tests to allow access to some well known pre-determined resource identifier, interact with the gRsrcMngr to retrieve the resource, assume the resource is ready to use, and start using it for its own purposes. A good example of this is to allow lower numbered tests to create the ASQ/ACQ resources and assign some well known identifier like "ASQ" and "ACQ" to them. Higher numbered tests would be made simpler because now they can just use the ASQ/ACQ immediately. Group lifetime effectively allows code reuse in all tests cases contained within a single group. This also allows the building up of highly functional test scenarios by leveraging an understanding of prior tests. This power also has a downside in that we can no longer just choose to run any given test within any group and expect it to pass. We would have to run the tests which setup the environment for tests which have prerequisites. The framework does not understand these inherent prerequisites and so this task is a manual endeavor. However one can run an entire group and expect it to pass, because groups of tests must guarantee coherency with respect to each other.
The gRsrcMngr has a niche, it does one thing really well; it creates objects/resources with group lifetime. However this comes at the cost of the overhead of having to generate a call to it and supply and track an identifier for each resource. A test may desire to create a resource for temporary, localized use. These resources are said to have test lifetime. Any resource created with test lifetime, which is constructed by using a local function variable or a class level member variable will be destroyed/freed/deallocated after the test completes. This is ensured, because the framework deletes each test object after it completes, regardless if it completed in success or failure. The deletion of a test destroys all contained resources if and only if proper guidelines are followed. The risks associated with not following these subsequent recommendations could yield complex memory leaks.
This design allows, but highly shuns upon the use of malloc()/free(), and new/delete operators. To make your code more likely to become accepted back into nvmecompliance it is recommended not to use these mechanisms. However the framework doesn’t safeguard nor prevent their use. If they must be used then simply follow safe programming practices by coding the corresponding free() or delete in the destructor or wherever appropriate. Be aware that this mechanism does not work well with lifetime promotions. In other words, if a variable is created with test lifetime using malloc() and then assigned to a resource with group lifetime, lifetime promotion will not occur and a segmentation fault will most likely result. This design attempts to guide you in the right direction, but does not bind you to it. Attempts to fly your own route south for the winter carry the risk of losing technical guidance and support.
For proper test lifetime support, this design promotes the use of smart pointers. Each and every resource contained within the framework will provide members and typedefs to aid in smart pointer, i.e. specifically shared_ptr, usage. The concept of a shared_ptr is that the pointer owns the resource. As long as there is a shared_ptr pointing to a resource, that resource is guaranteed to exist. Declaring a local function variable or a class level member variable of type shared_ptr will be deallocated correctly when the test completes. Each test instance is destroyed by the framework when it completes, which destroys the shared_ptr, and thus destroys the object to which it points. This approach also works seamlessly with lifetime promotion. A resource with test lifetime will be correctly promoted to group lifetime if a shared_ptr is utilized.
There is 1 gotcha with using any type of pointer with respect to enforcing test lifetime. Test lifetimes are enforced by the framework by using a customized cloning technique. Objects/resources are cloned, then freed/deallocated after each test completes to force resource cleanup. This cloning uses special copy construction and operator=() to achieve proper resource cleanup. The end result of a clone action should be to re-create a new object without doing any copying of member variable pointers. Do not do shallow or deep copies of any pointer. Any pointer is one of the following: shared_ptr, weak_ptr, or C++ pointers using the new operator or malloc(). All newly cloned resources must achieve NULL member variable pointers or resource cleanup will not be correctly performed. This means default copy constructors and operator=() cannot be used because they do bitwise copying. If a child does not supply explicit over loaded methods for cloning, then the compiler inserts default methods and performs bitwise copying. Thus the absence of these methods in all children will cause memory leaks. It should be noted that creating resources with test lifetimes in the form of local pointers within member functions do not suffer this problem. Cloning only affects pointers declared with class level scope. If class level pointers, i.e. member variable pointers, are desired, rethink your strategy to perhaps pass the pointer to your functions, rather than have those functions access the test object members. If class level pointers are necessary, then assign those pointers = NULL within the copy constructor and operator=() will be sufficient. The gotcha comes when you forget to do this, or you forget to explicitly define a copy constructor or operator=() for your child test class. This gotcha is lurking for all children of class Test no matter how far down the inheritance chain they may live.
A resource created with test lifetime can be promoted to group lifetime status only when the test lifetime resource is associated with a resource with group lifetime. This occurs automatically when a shared_ptr is used. The uses of other types of pointers are not supported.
A good example of lifetime promotion is when an IOSQ, i.e. I/O Submission Queue, is created using the gRsrcMngr. The IOSQ now has group lifetime scope. The intent here is to allow subsequent tests to use this resource. After an IOSQ is born it must be initialized. This is when and how you will want to specify the type of memory backing this resource. If you choose to back this resource with discontiguous memory, then you must allocate this memory in user space and associate that with the IOSQ. The memory is itself a resource and thus can be created with group or test lifetime scope. Let’s say you decide to create resource MemBuffer using the recommended shared_ptr but do so within a local function. Thus this MemBuffer resource now has test lifetime. You then proceed to initialize the IOSQ using the localized MemBuffer resource.
Have we now created a problem? Because obviously when the test completes, the shared_ptr within the function, which owns the MemBuffer, will go out of scope and so doesn’t this imply the object it points to will also be deleted? Moreover, if the MemBuffer object becomes deleted, which is backing the IOSQ, doesn’t this imply that any subsequent test using it will cause a segmentation fault?
The answer is NO, because shared_ptr’s are used everywhere within the framework. So when you initialized the IOSQ with the MemBuffer it became assigned to a shared_ptr within the IOSQ. At this point we now have 2 shared_ptr’s within tnvme pointing to a single resource. When the test object’s shared_ptr gets deleted, the shared_ptr within the IOSQ will live on and so the MemBuffer is said to be “promoted” to group lifetime.
Proper smart pointer, i.e. shared_ptr, best practices are defined by the boost library to make use of the new operator but not the delete operator. The shared_ptr takes care of calling delete on your behalf because it is said that the pointer owns the object/resource after it is allocated from the heap.
Most resources have components at all levels of this design. References to a resource can exist from the DUT, from dnvme and from tnvme simultaneously. If a test happens to disable the DUT while resources are currently allocated, depending upon the type of resource, those resources may no longer exist with respect to the DUT. For example, the action of disabling a DUT causes the DUT to forget knowledge of all IOQ’s. dnvme actually deallocates all memory under its control when disabling occurs to mimic the DUT’s responsibilities in kernel space. However all the disjoint resources residing within tnvme need to be notified of this devastating action to allow proper cleanup all the way up the call chain.
This is where the subject-observer design pattern comes into play. The gRsrcMngr observes another global singleton called the gCtrlrConfig, i.e. Controller Configuration. The gCtrlrConfig facilitates the support for disabling and enabling a DUT. gCtrlrConfig is the subject which needs observation. The gRsrcMngr observes gCtrlrConfig because it needs to know when the DUT has been disabled. Each time the DUT is disabled the gRsrcMngr is notified and immediately deallocates all objects under its control. It would be a mistake for a test to use a resource which has been born of the gRsrcMngr after the DUT has been disabled. We see then, that group lifetime objects are handled by the framework on behalf of the test, however that still leaves test lifetime objects still allocated. Since the test is the one which is doing the disabling, then the test is the one which must understand that using localized test lifetime objects after a DUT has been disabled could actually cause a core dump in the worst case scenario. The test does not have to do any deallocating, that will be handled by the framework when the test completes. The test only needs to understand not to use any resources after it disables a DUT.
There is one exception to this rule. The design allows disabling a DUT in 1 of 2 ways, which will be completely destructive or only partially destructive. The complete destruction of all resources was previously discussed and occurs when a test calls gCtrlrConfig->SetState(ST_DISABLE_COMPLETELY). The less destructive way to disable a DUT is performed by calling gCtrlrConfig->SetState(ST_DISABLE). This later invocation destroys all objects except an ASQ and an ACQ. These objects will remain regardless if they are born with group lifetime or test lifetime, with the caveat that test lifetime admin Q’s will not be usable after the test completes. Remember test lifetime objects are deallocated after a test completes. dnvme will not modify the value of the DUT’s AQA, ASQ, and ACQ registers during the ST_DISABLE invocation. Any value those registers contained before ST_DISABLE will remain when the DUT is later enabled. Hence, any dnvme or tnvme resources backing an ASQ or an ACQ must also remain. The lesson here is that a test can invoke a ST_DISABLE, then ST_ENABLE and then immediately start using the ASQ or ACQ resources like nothing happened. All other resources however, will be deallocated.