-
Notifications
You must be signed in to change notification settings - Fork 99
Interrupts
The details of incorporating interrupts into this design mostly resides within dnvme, but the high level details will be discussed here. IOQ’s may or may not choose to use interrupts to notify of CE’s arriving from a DUT. If interrupts are not used then the host software must poll those CQ’s for new entries by observing the P-bit of the CE’s pointed after the head pointer. Moreover, a concurrent mixture of support must be handled where some CQ’s choose to and others do not use interrupts. The framework handles both schemes simultaneously by adopting a polling interface to the kernel and attaching ISR support so that both mechanisms report when CE’s arrive correctly. In fact, if ISR support is active for a particular CQ, the interface to those CQ’s looks just like the CQ is operating under a polling scheme. In other words, a test case developer does not have to customize coding for each CQ dependent upon what type of reporting scheme is active.
Reaping CE’s from a CQ is done by 1st checking if a particular CQ has any elements to reap. This is done non-intrusively by calling any one of the many CQ::ReapInquiryXXXX() methods from a queue resource object.
CQ::ReapInquiry()
This method returns immediately with an answer. There are either CE’s waiting or not and the number which are available to reap is returned. This method will not wait, and if a test case calls this method before a DUT has had a chance to process a previously submitted command, then returned value could very well be 0.
CQ::ReapInquiryWaitAny()
This method will return when any number of CE’s have arrived in the CQ or a timeout occurs, whichever comes first.
CQ::ReapInquiryWaitSpecify()
This method will return when at least a specified number of CE’s have arrived in the CQ or a timeout occurs, whichever comes first.
As long as at least 1 CE is waiting in a CQ, it can be reaped. Reaping from a CQ with 0 CE’s waiting is NOT considered an error, it's simply unproductive. Reaping does not imply that all CE’s must be reaped, and it does not mean that any CE’s need to be reaped. CE’s can be left in a CQ as long as a test desires, but if they are reaped then must be reaped in order of placement. The reaping action requires a test case to allocate enough memory to receive those CE’s which are being reaped. If too little memory is allocated, this is not an error, dnvme will return as much as it can up to and including the maximum number of CE’s requested. After a reap, a subsequent call to ReapInquiry() will not be needed, because the act of reaping returns the number of CE’s remaining in the CQ. The CE’s which are successfully reaped will be reported back to tnvme. dnvme will also update the appropriate doorbell register by specifying the number of CE’s successfully reaped back to tnvme. It is important to realize that nothing will be done automatically; action onto the DUT is incurred by dnvme only when the direct authorization arrives from tnvme.
This interface conforms nicely to the polling scheme, for this is the definition of polling. CQ’s are polled until items arrive and then they may be reaped if need be.
By default all interrupts are disabled when tnvme launches. This includes disabling interrupts for the ACQ and thus polling the ACQ will be the only way to extract CE’s from it. The disabling of pin-based, MSI-single, MSI-multi and MSI-X is supported. The enabling of all types of interrupts is supported except pin-based. If pin-based support is desired, a hybrid approach will need to be invoked by writing custom code in dnvme. Pin-based interrupts don’t fit nicely into this design, however all other interrupts do.
To enable interrupts a test developer must interact with CtrlrConfig::SetIrqScheme(). This method sets the desired scheme for the entire DUT to one of MSI-single, MSI-multi or MSI-X. In order to set/change the active scheme the DUT must be disabled. This scheme becomes active when the DUT is enabled and remains until the DUT is again disabled. Moreover, the framework disables all interrupts before each group initiates execution. Remember the framework attempts to bring the system back to a known state before each group executes so that each effectively starts from the same state. This also implies that the DUT is disabled before each group executes, because in order to disable interrupts this is a prerequisite.
After interrupts are setup, the ACQ automatically, and must, start to use interrupt vector 0 according to the NVMe specification. dnvme will automatically create the meta data within the kernel so that when a test gets around to creating an ACQ resource within tnvme, it will already be able to start using interrupts. In other words, if interrupts are enabled and an ACQ exists, then its CE's must be reaped using an interrupt scheme from the DUT. As IOCQ's are created by a test, they choose to use the active interrupt scheme or not during initialization, reference method CQ::Init(). If an IOCQ decides to use interrupts, it must supply the interrupt vector to associate.
When enforcing an IOCQ to use interrupts, what is the set of available vectors which can be supplied during initialization? This will be the set of vectors from 0 to (n-1), where n will be specified to a call to CtrlrConfig::SetIrqScheme().
After an IOCQ has been initialized to use interrupts, the 2nd half of creation must occur by associating that IOCQ resource with a Create IOCQ command, and then issuing the command to a DUT. After the command completes successfully and is reaped from the ACQ, the IOCQ resource can be used, but its CE’s must be reaped using the chosen interrupt scheme just like for the ACQ.
When a CQ must use interrupts, how does this effect tnvme? It affects the coding of tnvme is no way what-so-ever. In fact, the same methods describe above for polling are the identical methods used for a CQ implementing interrupts. The difference comes in how dnvme handles the IOCTL’s which support the reap inquiry and reap requests from tnvme.
Recall that reap inquiry returns the number of CE’s in a given CQ at some point in time. If polling is in effect, a reap inquiry request causes dnvme to seek through the CQ inspecting the P-bit for the CE’s coming after the head pointer. If an interrupt is in effect, a reap inquiry request causes the same algorithm to be run against the CQ, if and only if, that CQ has registered an interrupt arriving from the DUT. Remember the only information being transferred from the DUT to the kernel is that something has arrived in the CQ associated with a particular interrupt vector. This design conforms to this transfer of knowledge by returning 0 CE’s residing within a CQ, even though the DUT may have actually placed CE’s there, if the DUT never fired the appropriate interrupt vector. Thus if 0 has been returned, then none may be reaped. This conforms to the NVMe specification, because a kernel, i.e. any NVMe driver, would not have been notified of any CE’s in a CQ, and thus they would not have been reaped, because the interrupt vector never fired. Both in this test environment and in the real world, no CE’s can be reaped without a properly operating DUT invoking the correct interrupt vector.
If the correct IRQ is signaled, and tnvme learns that a number of CE’s are waiting in a CQ, what would happen if a test only reaped a partial number of those CE’s? Or worse yet what happens when more CE’s arrive while tnvme hasn’t reaped all the CE’s from the prior IRQ? There are a number of questions which may be raised, but all of them could also be raised outside of this test environment in a production level driver. In other words, all of these situations are also possible with any driver, not just dnvme. It may be beneficial to review how MSI based IRQ’s work by looking at the table in the NVMe specification in section 7. Remember pin-based IRQ’s are not supported because they don’t fit nicely into this design, and after reviewing that table you may come to understand why. In a nutshell that table outlines the behavior of MSI based interrupts in any situation we may ponder. In the two afore mentioned questions the behavior will be dictated by the appropriate active IRQ scheme in play. For further details see MSI-X Interrupts or MSI Interrupts.