[virtio-pmem] Implementation #5463

ShadowCurse · 2025-10-03T18:49:15Z

Changes

Add virtio-pmem device support.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

codecov · 2025-10-03T18:58:50Z

Codecov Report

❌ Patch coverage is 72.97297% with 140 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.62%. Comparing base (f335a0a) to head (7d83503).
⚠️ Report is 3 commits behind head on main.

⚠️ Current head 7d83503 differs from pull request most recent head 9a554b4

Please upload reports for the commit 9a554b4 to get more accurate results.

Files with missing lines	Patch %	Lines
src/vmm/src/devices/virtio/pmem/device.rs	67.67%	64 Missing ⚠️
src/vmm/src/devices/virtio/pmem/event_handler.rs	23.07%	40 Missing ⚠️
src/vmm/src/builder.rs	77.14%	8 Missing ⚠️
src/vmm/src/rpc_interface.rs	0.00%	8 Missing ⚠️
src/firecracker/src/api_server/request/pmem.rs	77.27%	5 Missing ⚠️
src/vmm/src/devices/virtio/pmem/metrics.rs	85.29%	5 Missing ⚠️
src/vmm/src/resources.rs	58.33%	5 Missing ⚠️
src/vmm/src/vmm_config/pmem.rs	93.33%	3 Missing ⚠️
src/vmm/src/device_manager/persist.rs	96.55%	1 Missing ⚠️
src/vmm/src/devices/virtio/pmem/persist.rs	95.23%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5463      +/-   ##
==========================================
- Coverage   82.79%   82.62%   -0.17%     
==========================================
  Files         263      269       +6     
  Lines       27226    27729     +503     
==========================================
+ Hits        22541    22911     +370     
- Misses       4685     4818     +133

Flag	Coverage Δ
5.10-m5n.metal	`82.77% <72.97%> (-0.20%)`	⬇️
5.10-m6a.metal	`82.02% <72.97%> (-0.20%)`	⬇️
5.10-m6g.metal	`79.43% <72.97%> (-0.15%)`	⬇️
5.10-m6i.metal	`82.76% <72.97%> (-0.22%)`	⬇️
5.10-m7a.metal-48xl	`82.01% <72.97%> (-0.20%)`	⬇️
5.10-m7g.metal	`79.44% <72.97%> (-0.14%)`	⬇️
5.10-m7i.metal-24xl	`82.74% <72.97%> (-0.21%)`	⬇️
5.10-m7i.metal-48xl	`82.74% <72.97%> (-0.20%)`	⬇️
5.10-m8g.metal-24xl	`79.43% <72.97%> (-0.14%)`	⬇️
5.10-m8g.metal-48xl	`79.43% <72.97%> (-0.15%)`	⬇️
6.1-m5n.metal	`82.79% <72.97%> (-0.21%)`	⬇️
6.1-m6a.metal	`82.05% <72.97%> (-0.20%)`	⬇️
6.1-m6g.metal	`79.43% <72.97%> (-0.15%)`	⬇️
6.1-m6i.metal	`82.80% <72.97%> (-0.21%)`	⬇️
6.1-m7a.metal-48xl	`82.04% <72.97%> (-0.20%)`	⬇️
6.1-m7g.metal	`79.44% <72.97%> (-0.14%)`	⬇️
6.1-m7i.metal-24xl	`82.81% <72.97%> (-0.20%)`	⬇️
6.1-m7i.metal-48xl	`82.81% <72.97%> (-0.20%)`	⬇️
6.1-m8g.metal-24xl	`79.43% <72.97%> (-0.14%)`	⬇️
6.1-m8g.metal-48xl	`79.44% <72.97%> (-0.14%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

msync is used by virtio-pmem device to trigger sync of mmaped file content to the underlying file. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Manciukic

we should update docs/device-api.md.
changelog entry
any performance tests? we could check how fast we can read or write the entire pmem or maybe we can integrate it with the block tests using fio

Manciukic · 2025-10-07T10:32:51Z

src/vmm/src/devices/virtio/pmem/device.rs

+                libc::MS_SYNC,
+            ) < 0
+            {
+                result = FAILURE;


we should log the error

Manciukic · 2025-10-07T10:33:43Z

src/vmm/src/devices/virtio/pmem/device.rs

+    fn interrupt_trigger(&self) -> &dyn VirtioInterrupt {
+        self.device_state
+            .active_state()
+            .expect("Device is not implemented")


nit: "is not active"?

Manciukic · 2025-10-07T10:34:37Z

src/vmm/src/devices/virtio/pmem/device.rs

+        }
+    }
+
+    fn write_config(&mut self, _offset: u64, _data: &[u8]) {}


we could log unexpected attempts to write

we don't log such writes in any device, so don't think we should do it here.

Manciukic · 2025-10-07T10:43:42Z

src/vmm/src/vstate/vm.rs

+        let next = self.common.next_kvm_slot.load(Ordering::Relaxed);
+        if next == self.common.max_memslots {
+            None
+        } else {
+            self.common.next_kvm_slot.store(next + 1, Ordering::Relaxed);
+            Some(next)
+        }


to really be atomic we should do a fetch_add.

Manciukic · 2025-10-07T10:47:21Z

src/vmm/src/devices/virtio/pmem/persist.rs

+        pmem.avail_features = state.virtio_state.avail_features;
+        pmem.acked_features = state.virtio_state.acked_features;
+
+        pmem.set_mem_region(constructor_args.vm)?;


is the allocator state getting restored?

Manciukic · 2025-10-07T10:56:23Z

docs/pmem.md

+      "id": "pmem0",
+      "path_on_host": "./some_file",
+      "root_device": true,
+      "read_only": fasle


- "read_only": fasle + "read_only": false

Manciukic · 2025-10-07T10:56:57Z

docs/pmem.md

+         \"root_device\": true,
+         \"read_only\": false
+    }"
+```


we should probably mention snapshot/restore behaviour as well

and also security considerations about sharing memory (which we do not recommend).

We can also mention performance considerations: ie that even though pages are in memory, the guest still needs to exit to the kernel to set up the pagetable mappings. Using hugetlbfs to back the file would be faster (but will consume memory).

added snapshot, security and performance sections

Manciukic · 2025-10-07T11:01:51Z

src/vmm/src/devices/virtio/pmem/device.rs

+        }
+        let mmap_len = align_up(file_len, Self::ALIGNMENT);
+
+        let mut flags_1 = libc::MAP_SHARED | libc::MAP_ANONYMOUS | libc::MAP_NORESERVE;


PRIVATE?

nit: having the flags inline with the syscall would make it more readable

Manciukic · 2025-10-07T11:08:05Z

tests/integration_tests/functional/test_pmem.py

+    assert False
+
+
+def test_pmem_add(uvm_plain_any, microvm_factory):


should we check writes are persisted to disk?

Manciukic · 2025-10-07T11:10:48Z

docs/pmem.md

+> `DAX` support is not uniform for all file systems. Check the documentation for
+> the file system you want to use before enabling `DAX`.


it works on ext4, right? does it need any specific options (ie 4096 block size) or just works?

Add implementations of device, event handling, metrics. Add device config and builder types for API use. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Update VmResources type with virtio-pmem configuration field to allow virtio-pmem devices be configured through config files and later through API calls. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Both virtio-block and virtio-pmem can act as root devices for a VM. Add a check to prevent specifing more than 1 root device for a VM. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add /pmem/id PUT request for virtio-pmem configuration. Add corresponding metrics. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Virtio-pmem devices need to allocate a memory region in guest physical memory. The safe place to do this is past 64bit MMIO region. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add a counter for KVM slot ids into VmCommon struct. This is done because virtio-pmem device needs to obtain it's KVM slot id independently from number of slots in GuestMemoryMmap. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add methods to attach virtio-pmem devices to Vmm. Add methods to create KVM memory slot for virtio-pmem devices. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add logic to store and restore virtio-pmem device information in a snapshot. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add functional and API tests for virtio-pmem device and its configuration fields Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Expose virtio-pmem metrics in the logger, so they are exported in metrics.json. Update integration tests to expect new metrics. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add description of pmem endpoint. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add new document about virtio-pmem configuration and usage. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add a note about addition of virtio-pmem device. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse self-assigned this Oct 3, 2025

ShadowCurse force-pushed the virtio_pmem branch 2 times, most recently from d8f9547 to 5970613 Compare October 6, 2025 11:01

ShadowCurse marked this pull request as ready for review October 6, 2025 12:57

ShadowCurse added Status: Awaiting review Indicates that a pull request is ready to be reviewed Type: Enhancement Indicates new feature requests labels Oct 6, 2025

ShadowCurse force-pushed the virtio_pmem branch 2 times, most recently from a8bedbb to 1d2aeb2 Compare October 6, 2025 14:18

ShadowCurse requested review from bchalios and Manciukic October 6, 2025 14:22

chore(virtio-pmem): add msync syscall to seccomp filters

eb0fa41

msync is used by virtio-pmem device to trigger sync of mmaped file content to the underlying file. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the virtio_pmem branch from 1d2aeb2 to 7d83503 Compare October 7, 2025 08:43

Manciukic reviewed Oct 7, 2025

View reviewed changes

ShadowCurse added 11 commits October 7, 2025 12:26

feat(virtio-pmem): add device configs and implementation

ee55ddc

Add implementations of device, event handling, metrics. Add device config and builder types for API use. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat(virtio-pmem): add device to the VmResources

92e231e

Update VmResources type with virtio-pmem configuration field to allow virtio-pmem devices be configured through config files and later through API calls. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat: add a check for a single root device

773cf35

Both virtio-block and virtio-pmem can act as root devices for a VM. Add a check to prevent specifing more than 1 root device for a VM. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat(virtio-pmem): add API requests parsing to Firecracker

bc6152f

Add /pmem/id PUT request for virtio-pmem configuration. Add corresponding metrics. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat: add allocator for past 64bit memory region

ecd0cf7

Virtio-pmem devices need to allocate a memory region in guest physical memory. The safe place to do this is past 64bit MMIO region. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat: add a counter for KVM slots

97e6ba8

Add a counter for KVM slot ids into VmCommon struct. This is done because virtio-pmem device needs to obtain it's KVM slot id independently from number of slots in GuestMemoryMmap. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat(virtio-pmem): add device to the Vmm

03149be

Add methods to attach virtio-pmem devices to Vmm. Add methods to create KVM memory slot for virtio-pmem devices. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat(virtio-pmem): add snapshot support

adf3555

Add logic to store and restore virtio-pmem device information in a snapshot. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat(virtio-pmem): add integration tests

3b54b4e

Add functional and API tests for virtio-pmem device and its configuration fields Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat(virtio-pmem): export device metrics

f9f23ac

Expose virtio-pmem metrics in the logger, so they are exported in metrics.json. Update integration tests to expect new metrics. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

chore(virtio-pmem): update swagger file

085543b

Add description of pmem endpoint. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the virtio_pmem branch from 7d83503 to efd93ea Compare October 7, 2025 12:48

ShadowCurse requested review from xmarcalx, kalyazin and pb8o as code owners October 7, 2025 12:48

doc(virtio-pmem): add documentation

9e690fd

Add new document about virtio-pmem configuration and usage. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

chore: update CHANGELOG

9a554b4

Add a note about addition of virtio-pmem device. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the virtio_pmem branch from efd93ea to 9a554b4 Compare October 7, 2025 12:49

		assert False


		def test_pmem_add(uvm_plain_any, microvm_factory):

		> `DAX` support is not uniform for all file systems. Check the documentation for
		> the file system you want to use before enabling `DAX`.

[virtio-pmem] Implementation #5463

Are you sure you want to change the base?

[virtio-pmem] Implementation #5463

Uh oh!

Conversation

ShadowCurse commented Oct 3, 2025

Changes

License Acceptance

PR Checklist

Uh oh!

codecov bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Oct 3, 2025 •

edited

Loading