Queues

Queues are resources, but only the ASQ, ACQ, IOSQ and IOCQ classes are meant to be instantiated by tests. A queue resource residing within tnvme represents the corresponding physical entity residing within dnvme. However, tnvme has complete control over those resources. For SQ’s tnvme controls what, and when commands are placed within it and when its doorbell is rung. For CQ’s, tnvme controls when and if items are reaped from CQ’s, and how many elements to reap per reap request. It is important to understand that dnvme will never do anything to any queue unless specifically told to by tnvme. The interaction between tnvme and dnvme occurs via IOCTL’s, but these details are obscured from a test developer on purpose.

The framework provides queue classes for tests to instantiate and initialize. A test can interact with a queue resource at its leisure. A test simply calls the public methods of the class hierarchy to control those resources. A common scenario might be for a test to request that the gRsrcMngr allocate an object and then typecast the resulting object into the appropriate shared_ptr. A good place to seek for code examples is to peruse the source code files within directory tnvme/GrpBasicInit. After a queue object is born it must be initialized for it to be useful. Simply call one of the resources public Init() methods. During initialization is where tnvme notifies dnvme what type of memory is to back this resource.

Backing a queue resource with memory requires a test to call the appropriate public Init() method of that resource. There are currently two methods and they allow this backing using discontiguous or contiguous memory, but not both at the same time. Discontiguous memory backing requires tnvme to allocate a MemBuffer resource and specify the alignment requirements for it to consume. Contiguous memory backing is only possible by authorizing dnvme to allocate the memory on tnvme’s behalf and subsequently mmap that back to user space as r/w memory. Hence, no matter what type of memory is backing a queue resource, tnvme has the ability to peek into it. An important note here is to point out that peeking into queue memory is the proper way to reference it, never should tnvme directly write into memory backing a queue. The framework is not meant to allow this, and doing so will most certainly cause uncertain behavior. The framework attempts to guard against writing to memory backing a queue by providing public assessors which return a const memory pointer. But often enough there are ways to circumvent all software constraints..

After a queue resource has been initialized it can be used immediately if it is an ASQ or ACQ. Of course the controller must first be enabled. However, if the queue object is an IOQ type, then an admin command must be sent to the DUT which allows the 2nd half of creation to proceed. If this creation command succeeds, then tnvme may use it to issue commands. Otherwise do not send commands because it wasn’t really created. In fact, dnvme cleans up all allocations associated with an unsuccessful creation when that command is completed, successfully or in failure, by a DUT, but not until tnvme reaps the Completion Element (CE) from the CQ. If the 2nd half of creation is unsuccessful, you will have to start all over by reinitializing the queue object if you intend on using it again.

The different queue types have different public assessors depending upon the role they play and what makes sense to allow. For example, the proper way to write into a SQ is to send a command to it by calling public method SQ::Send(). CQ’s should only ever be written to by the DUT itself. tnvme must only ever reap elements from CQ’s by 1st calling any of the CQ::ReapInquiry() methods to learn if there are elements to be reaped. Subsequently calling CQ::Reap() will physically reap as many items from the CQ as you so desire, you are not obligated to reap all elements.