vmm: add pci_segment mmio aperture configs

[thomasbarrett]

When using multiple PCI segments, the 32-bit and 64-bit mmio aperture is split equally between each segment. Add an option to configure the 'weight'. For example, a PCI segment with a mmio32_aperture_weight of 2 will be allocated twice as much 32-bit mmio space as a normal PCI segment.

Alternatives Considered

I considered adding a mmio32_aperture_size config instead of mmio32_aperture_weight config to precisely configure the size of the mmio apertures for each segment. I chose to not implement this because the logic to implement that correctly was a lot more complicated. Also, allocating mmio space between PCI segments does not actually require that level of precision.

Use Case

Unfortunately, some devices actually use quite a bit of 32-bit MMIO space. For example, a NVIDIA NVSwitch device uses 32 MB of 32-bit mmio. When using a few PCI segments and a few NVSwitch VFIO devices, you can quickly overwhelm the default 32-bit mmio allocation for a PCI segment.

Example

The following example allocates 2 PCI segments. The first segment is allocated 1/4 of the available mmio32 space. The second segment is allocated 3/4 of the available mmio32 space.

--platform pci_segments=3
--pci-segment pci_segment=1,mmio32_aperture_weight=1
--pci-segment pci_segment=2,mmio32_aperture_weight=3

Backward Compatability

This is completely backwards compatible. By default, each PCI segment has a aperture weight of 1.

[rbradford]

I guess this is fine (the code and solution are excellent - i'm just always loathe to increase our feature set / API interface as there is more to test and maintain). When adding the segment support I considered only supporting devices with 32-bit BARs on segment 0 - unfortunately the experience would be frustrating as there is no way to know a device needs 32-bit bars until we call allocate_bars which is relatively late in the device initialisation.

What about testing? Could do some unit tests - if you extracted a function create_mmio_allocators then that could be unit tested - config changes should be unit tested too. Any scope for integration testing?

[up2wing]

The code LGTM.
Maybe it's better to add some description in docs/iommu.md.

[thomasbarrett]

Now that the solution / UI has high level approval, I will go back today and add unit tests, integration tests, and documentation.

[thomasbarrett]

Okay, I have added:

1. unit tests for both create_mmio_allocators.
2. unit tests for the config additions.
3. documentation for the new config additions.

I can't think of a good integration test. If anyone has suggestions, I would be happy to add. Otherwise, I think this is g2g.

[rbradford]

I'm happy with unit tests - we should have more of those - we should aim to have more of those - they run much quicker and can also help structure the code for readability.

[likebreath]

LGTM. Only couple of nits about documentation formatting. Thank you for the contributions.