VMs with more than 254 vCPUs

[iximeow]

Going up to 254 vCPUs is straightforward enough, and with a recent Helios change we're free to bump Propolis' MAXCPU up to 254, but going beyond that will involve some work.

The obvious and somewhat daunting solution would be for Propolis to grow an IOMMU which can be used to let guests remap interrupts towards high-number cores like you'd do on real hardware.

https://david.woodhou.se/ExtDestId.pdf describes an alternative that might be workable: Hyper-V, KVM, FreeBSD byhve, and Xen all support a hypervisor feature to expand APIC IDs as used in MSI, but guest OS support is more mixed. Reportedly Linux supports this, Windows would be in a weird state either way, and the FreeBSD issue linked there is still open. I assume it would be mildly better for everyone to help get OSes in a good state with this than to build an IOMMU for Propolis, when the need comes..

[iximeow]

as I'm going through the ACPI table work in #999, the Processor Local APIC Structure definitions make clear that there is some work to do on the ACPI side too for many-processor VMs. probably?

the Local APIC structure has a one-byte field for APIC ID and for ACPI Processor UID. these clearly won't do when the number overflows. additionally, Local APIC NMI Structure has a one-byte ACPI Processor UID field as well. in both cases, ACPI points out that the Processor declaration operator is deprecated, that there's a compatibility note in the x2APIC forms of the declarations, and to read more there.

which tells me that in addition to however we're targeting interrupts (ExtDestId or an emulated iommu, as testing guides us) we'll want x2apic tested and working and to generate x2APIC-style processor definitions here.

[dancrossnyc]

Oh yes. It's painfully tedious, but we'll have to plumb the x2 versions of data for this stuff into the guests to get them to use more than 255 VCPUs. E.g., instead of (or in addition to) using the "Processor Local APIC Structure" of ACPI 6.6 sec 5.2.12.2, we'll have to supply the "Processor Local x2APIC Structure" of sec 5.2.12.12.

I highly recommend using the enlightments Woodhouse mentioned in the links above; doing a virtualized IOMMU is going to be neither fun nor easy, even just for interrupt remapping, and all the guests we need to worry about should already support them.