Feature Request: Hypervisor Guest interrupts on APLIC-only systems

[konrad-schwarz]

Hypervisor Guest Interrupts in APLIC-only Systems

Rationale

There is a business case for multi-core SoCs without PCI-express but employing the RISC-V Advanced Interrupt Architecture (AIA), i.e., look at the many multi-core SoCs without PCI-express targeting low-end embedded systems in existence today.

The AIA addresses these systems with an APLIC-only solution. As such systems are basically not modifiable in the field, a simple wired interrupt request scheme, such as provided with the APLIC, suffices.

A useful software architecture for these devices is to use a simple hypervisor to statically partition the multi-core into different "cells" (consisting of subsets of the cores, of memory ranges, and of the system's devices the cells may access exclusively), with each running a separate operating system (such as Linux or an RTOS) or bare metal software. This sort of architecture occurs when consolidating legacy multi-board solutions on to a single SoC and can also be driven by safety concerns, where one would like to separate safety-critical code from the rest of the system as much as possible.

Unfortunately, the AIA in its present form does not allow such a hypervisor (footing on the RISC-V H-extension) to use the guest-interrupt passthrough feature on an APLIC-only implementation: this feature is supported by the Incoming MSI Controller (IMSIC) only. The interrupt passthrough feature allows interrupts to be passed through to guests (or cells) with no software overhead in the hypervisor, making it an attractive feature for real-time systems with tight timing constraints.

As an initial idea, I think it would suffice for each APLIC interrupt source to be augmented with an additional attribute, the virtual guest interrupt number, with an encoding identical to the hstatus.VGEIN field (a 6-bit field where the value 0 means non-guest, i.e., hypervisor). I do not expect the systems outlined above to require more than 63 cells in total. (A more general proposal would allow each interrupt source to specify its VGEIN value on a per hart basis.)

Implementation Suggestion

The target[i] registers in the MSI delivery mode already have a guest index field defined for this purpose when these bits are unused in the direct delivery mode. Using the same field for the same purpose in direct delivery mode would seem to satisfy the requirements of this request. On a hardware level, the cores in a SoC implementing only an APLIC would require an additional 6-bit bus from APLIC to cores that indicates the guest interrupt number when they implement the H-extension.

SEE API to APLIC

Incidentally, the APLIC suffers from the same security loophole as the PLIC: there is basically no way for M-mode code to protect its registers from S-mode (the PMP is not granular enough)[*]. The original PLIC proposal addressed this with a to-be-defined S-mode Execution Environment (SEE) API (e.g. an SBI extension) to deal with configuration of the PLIC, but this has not been realized for PLIC in openSBI to date.

[*] Assuming the memory address bus does not record the hart mode (M/S/U/VS/VU) of the access.

The APLIC chapter addresses these concerns when discussing the M-mode and S-mode interface to the IMSIC, but it applies to all registers: setip/setipnum/clipnum/setie/setienum/...

The same approach -- configuration via the SEE -- should be viable for the APLIC, but this time it needs to be done: for a simple system assuming non-antagonistic S-mode software, the present situation may be tenable, but for hypervisor systems, both large and small, security and safety are essential. With the RISC-V ISA explicitly acknowledging the existence of the SEE, a SEE API to the interrupt controller is an elegant way of not requiring fine grained PMP protection of APLIC registers on the one hand and to eliminate the need for trapping APLIC accesses or providing a new para-virtualized interface to the APLIC on the other: if the APLIC specification itself defines the API, hypervisors can implement that API as they see fit without affecting guest code bases (by requiring the use of new para-virtualized interfaces).

For completeness: a technically correct solution would be for the API to be specified as direct memory accesses to the registers themselves, with M-mode code decoding APLIC accesses in the memory-access exception handler and hypervisors decoding them in guest memory access exception handlers. If a technical case can be made for this solution, viz. para-virtualization via a new SEE API is not clearly more performant than trapping and decoding individual memory accesses, I guess it would be viable, but it should be made clear in the AIA specification that this step is necessary -- the comments regarding the interface registers to the IMSIC would then be redundant.

Perhaps a better solution would be to have the APLIC have a CSR interface instead of a memory-mapped one: an indirect interface using an address and a data register would probably suffice and reduce pressure on the CSR address space.