Skip to content

Run all tests on PCIe kernel #5300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 66 commits into
base: feature/pcie
Choose a base branch
from

Conversation

Manciukic
Copy link
Contributor

Changes

Move the PCI configuration to the VM factory so that we can have uvm_plain* fixtures also run the tests on PCIe kernels. Also cleanup some places where the uvm_plain was simpler.

Reason

Not all tests were running on PCIe, hiding some issues

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

  • I have read and understand CONTRIBUTING.md.
  • I have run tools/devtool checkstyle to verify that the PR passes the
    automated style checks.
  • I have described what is done in these changes, why they are needed, and
    how they are solving the problem in a clear and encompassing way.
  • I have updated any relevant documentation (both in code and in the docs)
    in the PR.
  • I have mentioned all user-facing changes in CHANGELOG.md.
  • If a specific issue led to this PR, this PR closes the issue.
  • When making API changes, I have followed the
    Runbook for Firecracker API changes.
  • I have tested all new and changed functionalities in unit tests and/or
    integration tests.
  • I have linked an issue to every new TODO.

  • This functionality cannot be added in rust-vmm.

bchalios and others added 30 commits July 4, 2025 10:40
This is just code organization changes. Create a new module under
`virtio`, called `transport`. For the time being the only transport
supported is `mmio`. Also, move `IrqInterrupt` type within the MMIO
transport code, as it is MMIO specific.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
`IrqTrigger::new()` returns a `Result` because creating an `EventFd`
might fail with an `std::io::Error` error. All users of `IrqTrigger`
create the object and directly unwrap the error.

To avoid unwraps all over the place, change `IrqTrigger::new()` to
unwrap a potential error while creating the EventFd internally and just
return `Self`.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
The MMIO transport for VirtIO devices uses an `IrqTrigger` object as the
object that models the logic for sending interrupts from the device to
the guest. We create one such object for every VirtIO device when
creating it. The MMIO device manager associates this object with an IRQ
number and registers it with KVM.

This commit changes the timing of association of an `IrqTrigger` with a
VirtIO-mmio device. It only assigns such an object to the device during
its activation. We do this to prepare for supporting a PCI transport for
VirtIO devices. The cloud hypervisor implementation for these passes the
interrupt objects used by the device during activation, so we make this
change to have a uniform way to handle interrupts for both transports.
Functionally, nothing changes for MMIO devices, as before activation we
don't trigger any interrupts.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Describing the APIs that need to implement types that are used as
interrupts for VirtIO devices. Currently, we only use `IrqInterrupt`
interrupts, but this will change once we have MSI-X with PCIe devices.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
VirtIO devices assume they're operating under an MMIO transport and as a
consequence they use IrqTrigger as interrupts. Switch that to using
VirtioInterrupt for all VirtIO device objects. Only assume a
VirtioInterrupt is an IrqTrigger in MMIO specific code.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Bring in the vm-device crate from CloudHypervisor. We will be using it
for adding PCIe support.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
We use `SerialDevice` with Stdin as the input source. Encode this in the
type so that we don't spill the generic all over the place.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Co-authored-by: Egor Lazarchuk <yegorlz@amazon.co.uk>
Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>
Signed-off-by: Babis Chalios <bchalios@amazon.es>
Use the vm_device::Bus bus for all MMIO devices. This is mainly to
prepare for using it for PCIe devices. Also, sepate VirtIO devices from
other MMIO devices inside the MMIODeviceManager struct. This makes
iterating over VirtIO devices easier since we don't need to access two
data structures to get a reference to a VirtIO device any more.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
We were always constructing RTCDevice using a set of metrics that were
defined in the RTC module itself. Don't leak the metrics to other
modules. Instead, create a new() function that always constructs it the
correct way.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Use the vm_device::Bus bus for PortIO devices on x86. PCIe devices will
use this as well.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
PCIe spec mandates that software can access the configuration space of
PCIe devices both via MMIO and Port IO accesses. As a result, PCIe
devices will need to register to both buses (on x86).

Change the organization of devices, so that MMIO and PIO device managers
do not own the buses. Instead, introduce a DeviceManager object which
holds the buses, the resource allocator and includes also all types of
device managers (at the moment MMIO, PIO and ACPI).

Signed-off-by: Babis Chalios <bchalios@amazon.es>
We always create anew the keyboard interrupt event. Just create it
inside `I8042Device::new()` and return an error if that fails.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
test_serial_dos test checks that when we send a lot of bytes in the
serial device the emulation logic does not increase indefinitely the
underlying buffer that we use for when the device is set in loopback
mode.

However, the test does not wait for the microVM to start and sometimes
the virtual memory allocation may increase between readings. Add a
network device to the microVM so that we implicitly wait until it has
booted before taking the first measurement.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Bring in pci crate from cloud hypervisor with a few modifications.
We use the rust-vmm vm-allocator crate instead of Cloud Hypervisor's
downstream one. For the time being, rust-vmm's implementation should
include all we need for supporting the devices we care about. If we need
more functionality from our allocators, we will implement the logic
directly in the rust-vmm vm-allocator crate.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
PCIe distinguishes MMIO regions between 32bit and 64bit, caring for
devices that can't deal with 64-bit addresses. This commit defines the
appropriate regions for both x86 and aarch64 architectures, extends the
resource allocator to handle allocations for both of these regions and
adjusts the logic that calculates the memory regions for the
architecture.

Also, un-do the change that added an `offset` argument
`arch_memory_regions` function. We won't be using this for "secret
hiding" so it just made the logic (especially for kani proofs) too
convoluted.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
PCIe devices need some times to relocate themselves in memory. To do so,
they need to keep an (atomic) reference to a type that implements
`DeviceRelocation` trait. The logic for relocation involves removing the
device from the bus it has been registered to, allocate a new address
range for it and reinsert it.

Instead of creating a new type for it, reuse `ResourceAllocator`. This
means that we need to move the buses from the `DeviceManager` inside
`ResourceAllocator`.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Add a PCIe segment which includes a single PCIe root port and a bus.

At the moment, the PCIe segment is always enabled. Later commit will
make it optional and enable it only when a command line argument flag is
passed to Firecracker binary.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
So that we can declare which memory region is used by PCIe devices for
MMCONFIG.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Write the PCI root bridge in FDT when PCI is enabled.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Add a command line argument to enable PCIe support. By default, PCIe
is disabled. The reason for making PCIe off by default is that users
need to explicitly enable PCI support in their kernels. Requiring users
to explicitly enable it, does not break existing deployments, i.e. users
can upgrade Firecracker within their existing environments without
breaking any deployment.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
At the moment, the logic just restores the device manager and add the
PCIe root complex if PCI is enabled.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Add an integration test that checks that `lspci` correctly locates the
PCIe root complex if PCI is enabled for the microVM. Also, add a
negative test that checks that PCIe root complex doesn't exist when PCI
is not enabled.

Also, extend coverage of, at least some of, the tests to ensure that
they run with and without PCI configuration enabled. Do that by
extending the `uvm_any*` fixtures to yield both variants.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
PCI-enabled guest kernels enable the `extd_apicid` CPU feature for AMD
CPU families after 16h. Our supported AMD families (Milan & Genoa) are
both 19h. This is irrespective of whether PCI is enabled in Firecracker.

Do not mark this as host-only when running with PCI enabled kernels,
i.e. all kernels that support ACPI.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
We have some Rust integration tests that check building and
booting of microVMs works correctly. Add variants for PCI-enabled
microVMs.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Tests test_spectre_meltdown_checker_on_guest and
test_check_vulnerability_files_ab run A/B tests between the HEAD of the
target branch and the tip of a PR branch. This will currently fail,
because Firecracker builds from the HEAD of the target branch know
nothing about the `--enable-pci` command line flag, so launching
the Firecracker binary for revision A will fail.

Only run these tests for non-PCI uVMs for now. Once this commit gets
merged we will re-enable and make sure that everything works as
expected.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
1. build the kernel with PCI/e support.
2. fix a race condition between udev renaming the network devices and
   fcnet setting up the network interfaces
3. install pciutils on the image

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
Signed-off-by: Babis Chalios <bchalios@amazon.es>
I've rebuilt the CI artifacts for the new development version.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
Signed-off-by: Babis Chalios <bchalios@amazon.es>
The memory monitor was only assuming a single MMIO gap on x86_64 when
calculating the memory regions that corresponded to guest memory. Now we
need to account for two MMIO gaps in x86 and one in ARM.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
When we re-arranged the MMIO address space in commit 9a165d1 (arch:
define 64-bit capable MMIO memory regions) we moved the MMIO region of
the boot timer device for x86 systems, but we didn't update the init
scripts that hardcode it and use it to report boot time timestamp back
to Firecracker.

Update the init.c and initramfs values for the region. Also, add a
functional test that runs during CI PR tests and makes sure the boot
timer works.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Commit be5a600 (tests: fix MMIO gaps in memory monitor tool) that fixed
the memory monitor to account for the 64-bit MMIO region included a
left-over debug print. Remove it.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
bchalios and others added 9 commits July 4, 2025 10:40
Add support for ITS device which provides support for MSI interrupts on
ARM architecture. This is currently supported only on systems with GICv3
interrupt controller.

In order to make saving/restore of ITS state work properly, we need to
change the order in which we restore redistributor register GICR_CTLR.
We need to make sure that this register is restored last. Otherwise,
restoring GICR_PROPBASER doesn't have any effect and ITS depends on it
in order to save/restore ITS tables to/from guest memory.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Refactor the test code that inserts VirtIO devices in a Vmm object and
then add a test which creates a Vmm with PCI devices and then serializes
and deserializes the device manager and ensures that everything is as
restored as expected.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Use pci_enabled fixture for boot time, block, and network tests to
create PCI microVM variants as well.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
We only pass pci=off if PCI is disabled in Firecracker. Adapt tests and
comments to reflect that.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
So that we don't have to downcast VirtioDevice trait objects to the
actual device type before calling the logic to process events for each
device.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Instead of storing internal allocators of ResourceAllocator within an
Arc<Mutex<>> container, just store `ResourceAllocator` itself in an
`Arc<Mutex<>>`.

Apart from that, we get rid of the `ResourceAllocatorState` state
object, and just clone `ResourceAllocator` itself when we want to
save/restore.

Also, make the creation of `ResourceAllocato` infallible, since we know
that the ranges we are using are correct.

Finally, fix saving/restoring the state of ResourceAllocator. We were
actually not resetting it correctly upon snapshot restore. The reason
why this was not a problem is that we don't actually need to perform any
new allocations post restore at the moment. However, like this we are
ready when we need to perform any hot-plugging operations. Also, add a
unit-test to ensure that this logic works correctly.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
We were confusing queue indexex with event indexes, when passing the
index of the queue that needed to be triggered after handling events.

Fix the logic to pass the correct index. This refactors a bit the code
to signal the queues in each event handler method. With MMIO we don't
need to signal each queue independently (one signal will cause the guest
to scan all queues). With PCI though, we are using MSI-X, so we need to
signal each queue independently.

Also, change vsock functional integration tests to also run for
PCI-enabled microVMs.

Signed-off-by: Babis Chalios <bchalios@amazon.es>
Correctly handle invalid Bdf by returning an error to the deserializer.

This bug was caught by the fuzzer.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
Add some unit tests to cover PciBdf parsing, conversion, and
(de)serialization.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
Copy link

codecov bot commented Jul 9, 2025

Codecov Report

Attention: Patch coverage is 30.26316% with 53 lines in your changes missing coverage. Please review.

Project coverage is 80.16%. Comparing base (e8c599c) to head (f1dd2b4).

Files with missing lines Patch % Lines
src/vmm/src/device_manager/mod.rs 0.00% 21 Missing ⚠️
src/vmm/src/lib.rs 0.00% 16 Missing ⚠️
src/vmm/src/persist.rs 0.00% 16 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff                @@
##           feature/pcie    #5300      +/-   ##
================================================
- Coverage         80.20%   80.16%   -0.05%     
================================================
  Files               265      265              
  Lines             30832    30867      +35     
================================================
+ Hits              24730    24745      +15     
- Misses             6102     6122      +20     
Flag Coverage Δ
5.10-c5n.metal 79.90% <26.38%> (-0.05%) ⬇️
5.10-m5n.metal 79.90% <26.38%> (-0.06%) ⬇️
5.10-m6a.metal 79.09% <26.38%> (-0.06%) ⬇️
5.10-m6g.metal 76.49% <30.26%> (-0.05%) ⬇️
5.10-m6i.metal 79.89% <26.38%> (-0.06%) ⬇️
5.10-m7a.metal-48xl 79.08% <26.38%> (-0.05%) ⬇️
5.10-m7g.metal 76.49% <30.26%> (-0.05%) ⬇️
5.10-m7i.metal-24xl 79.87% <26.38%> (-0.04%) ⬇️
5.10-m7i.metal-48xl 79.87% <26.38%> (-0.04%) ⬇️
5.10-m8g.metal-24xl 76.49% <30.26%> (-0.05%) ⬇️
5.10-m8g.metal-48xl 76.49% <30.26%> (-0.05%) ⬇️
6.1-c5n.metal 79.94% <26.38%> (-0.05%) ⬇️
6.1-m5n.metal 79.95% <26.38%> (-0.05%) ⬇️
6.1-m6a.metal 79.14% <26.38%> (-0.05%) ⬇️
6.1-m6g.metal 76.48% <30.26%> (-0.06%) ⬇️
6.1-m6i.metal 79.93% <26.38%> (-0.06%) ⬇️
6.1-m7a.metal-48xl 79.13% <26.38%> (-0.05%) ⬇️
6.1-m7g.metal 76.49% <30.26%> (-0.05%) ⬇️
6.1-m7i.metal-24xl 79.95% <26.38%> (-0.05%) ⬇️
6.1-m7i.metal-48xl 79.96% <26.38%> (-0.04%) ⬇️
6.1-m8g.metal-24xl 76.49% <30.26%> (-0.05%) ⬇️
6.1-m8g.metal-48xl 76.49% <30.26%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Manciukic Manciukic force-pushed the pcie/run-all-tests-on-pcie branch 3 times, most recently from a9e4ff3 to 3b558a7 Compare July 9, 2025 13:58
The vmm was only checking the mmio device manager for finding the device
to update. Use the generic device manager instead.

Also update unit tests that expect a specific string.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
@Manciukic Manciukic force-pushed the pcie/run-all-tests-on-pcie branch 2 times, most recently from be2d3c9 to c894a33 Compare July 9, 2025 14:15
The code managing the balloon logic is only looking at the mmio device
manager. Make it use the generic device manager to find the device.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
@Manciukic Manciukic force-pushed the pcie/run-all-tests-on-pcie branch 2 times, most recently from 9f992ac to fcaff05 Compare July 9, 2025 16:26
@Manciukic Manciukic mentioned this pull request Jul 10, 2025
10 tasks
@Manciukic Manciukic force-pushed the pcie/run-all-tests-on-pcie branch from 1999ba3 to 29ff20e Compare July 10, 2025 11:34
The device rename wasn't working on PCI devices because the code only
checked the MMIO state.
Fix the bug by looking for the device to rename in both the mmio and pci
states.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
@Manciukic Manciukic force-pushed the pcie/run-all-tests-on-pcie branch from 29ff20e to 104d1e1 Compare July 10, 2025 12:12
Currently, we're limited to 24 GSI lines, which is too little for PCI
devices. Keep the current ranges as "legacy IRQ", and create a new range
for "GSI" that goes up to the kvm theoretical maximum of 4096 lines.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
Tell systemd not to use "predictable names" for network devices (eg
enp0s1), but keep the ethN set by the kernel.
This is equivalent to passing net.ifnames=0 to the kernel command line.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
pci=off is just an optimization to skip the probing, it shouldn't matter
to the functionality of the tests. Dropping it to allow them to run with
PCI.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
Move the PCI configuration to the VM factory so that we can have
uvm_plain* fixtures also run the tests on PCIe kernels.
Also cleanup some places where the uvm_plain was simpler.

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
This patch makes 2 changes to make the test work on PCI:
 - simplify logic to find device address to be generic irrespective of
   ACPI/no-ACPI, PCI/no-PCI
 - move config offset from within the C program to the python test, as
   it's different between MMIO (0x100) and PCI (0x4000)

Signed-off-by: Riccardo Mancini <mancio@amazon.com>
@Manciukic Manciukic force-pushed the pcie/run-all-tests-on-pcie branch from 104d1e1 to f1dd2b4 Compare July 10, 2025 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants