SharedMemfd memory backend

Hello! This is Marco from the Firecracker team :)

We're designing a new set of APIs and a new SharedMemfd memory backend.

Together, they will allow nice things like pre- and post-copy migrations, backend-initiated snapshots that remove disk I/O for capturing the guest memory, live-migration in 10ms, and more!

The basic idea for SharedMemfd is that it's going to be an Uffd backend that also receives the memfd containing the guest memory during handshake. New APIs can tell it which pages in memory are dirty of have been discarded, so the backend has all the information it needs to read the guest memory.

Early documentation (subject to change): https://github.com/marco-marangoni/firecracker/blob/snapshot-improvements/docs/snapshotting/shared-memfd.md
Proof of concept with example backends is in the same branch.

The tentative release for the feature is the next Firecracker release for a subset of those APIs in dev-preview.

---

I'll be happy to hear your thoughts on this change, and especially which use-cases it can solve for you, and which use-cases are still not covered.

Thanks!

[WaylandYang]

Thanks @marco-marangoni — read the doc against what forkd ships (context: #5912 / #5951).

Covered. SharedMemfd replaces our MAP_SHARED patch, and snapshot_type: Backend replaces our vendored state-only snapshot type. Firecracker owning the memfd is fine for us.

Not covered: write-protect-based capture. Our live path doesn't pre-copy: pause → state dump → arm UFFDIO_WRITEPROTECT → resume → copy in the background, copy-before-unprotect on WP faults. The pause is independent of dirty rate and VM size (56 ms p50 on 1.5 GiB, almost all of it the state dump; numbers) and needs no dirty tracking.

Two questions:

1. Can a backend use WP on the handshake uffd? Its feature set is fixed by Firecracker at UFFDIO_API time, and a VMA can only belong to one uffd context, so we can't bring our own. Would you request the WP features when the host offers them, or register WP mode behind an opt-in?
2. Could the "must not be resumed until every pages_to_copy page is copied" rule be stated as the invariant instead (the snapshot holds each page as of the final create), so WP-based backends are within contract?

Happy to bench the snapshot-improvements branch and report back.

[marco-marangoni]

1. Right now we plan no changes to the UFFD protocol, i.e. only registering for MISSING/REMOVED pages. But your use-case is valid, so I'll think about that. I'm also thinking or allowing registering minor mode, so post-copy can be implemented with minor faults + UFFD_CONTINUE rather than UFFD_COPY, so the two changes could go hand in hand
2. Yeah, you're right: technically the invariant is that pages need to be copied with the content at the time of /snapshot/create. If we end up supporting WP, it would need to be rephrased

Thanks for taking a look!

[AdaAibaby]

Use-case: same-snapshot fan-out — cross-microVM copy-on-write page sharing (a memory-density goal, distinct from migration / live-fork)

Thanks @marco-marangoni — moving my closed #6380 here as you suggested. My use-case is a different axis from the migration and live-fork ones already in this thread, and I think it stresses exactly one assumption in the SharedMemfd design, so I want to state it precisely.
The goal
N microVMs resumed from the same snapshot (clone fan-out, RL rollouts, large same-template fleets) should share identical, untouched guest pages as copy-on-write, instead of each getting a private physical copy. For M = 4 GiB, N = 32, host memory for a template's live clones drops from ≈ N·M (~128 GiB) toward ≈ M + N·M·(1−r) as pages diverge (r = fraction still untouched, high early in a clone's life). It also removes the per-fault memcpy.
How SharedMemfd, as documented, maps onto this — and where it stops

• The memfd is allocated by Firecracker per instance (MemfdBacking, MAP_SHARED), and MAP_SHARED is there so the backend can read/write that one VM's memory for snapshot/migration. Faults are still MISSING, served by UFFDIO_COPY. So "shared" in SharedMemfd means Firecracker ↔ backend, for one VM — not physical pages across many VMs. Different axis.
• The minor-mode + UFFDIO_CONTINUE you floated (your 2026-10-13 reply) is the right bridge, but by itself it only covers single-VM post-copy: it saves the memcpy, yet each VM still faults against its own private memfd, so nothing is shared across VMs.
  The one real departure: backing ownership has to invert
  For cross-VM sharing the physical page can be shared only if all N VMMs map the same inode read-only MAP_SHARED, so the page cache holds one copy and UFFDIO_CONTINUE resolves each VM's minor fault against it. That means the backing can't be a memfd Firecracker allocates per instance (SharedMemfd's model) — it has to be a caller-provided, read-only, shareable backing (the snapshot memory file on shared storage, or a preloaded memfd handed in). Everything else in SharedMemfd stays; this is the single assumption that has to flip.
  Concretely, the minimal opt-in surface on the restore path:

1. Minor-mode registration on the restore uffd: request UFFD_FEATURE_MINOR_SHMEM / UFFD_FEATURE_MINOR_HUGETLBFS and register regions UFFDIO_REGISTER_MODE_MINOR (fall back to today's MISSING + anonymous where the kernel lacks it).
2. Caller-provided shared read-only backing, mapped MAP_SHARED instead of a per-instance private memfd — the inversion above.
3. Carry the registration mode in the handshake (GuestRegionUffdMapping) so the handler knows to use CONTINUE vs COPY.
4. Minor + WP (MODE_MINOR | MODE_WP): guest writes take a WP fault and diverge per-VM; untouched pages stay shared. This is also what @WaylandYang needs for WP-based capture (Allow MAP_SHARED for snapshot-restore mem_backend (cooperative-snapshot tooling) #5912 / [RFC] Allow MAP_SHARED for snapshot-restore mem_backend (cooperative-snapshot tooling) #5951 R3), so the two asks converge on the same "let the backend register minor/WP behind an opt-in" API — one surface serves both.
   Discard / isolation must be designed in, not bolted on
   With pages shared across tenants, pages_to_discard and any MADV_REMOVE / EVENT_REMOVE (balloon, free-page-reporting, virtio-mem) must act on the per-VM private overlay, never the shared read-only backing — otherwise one tenant's discard punches holes in pages other VMs are reading. Sharing must be strictly scoped to same-snapshot, read-only, copy-on-write pages, opt-in, off by default, with the isolation model documented. The MAP_SHARED memfd discard behaviour in bug: discard_range() silently fails to free physical memory for memfd-backed (MAP_SHARED) guest regions #6167 and AWS bulletin 2026-169 is exactly the territory this has to account for.
   Prior art
   The "map snapshot MAP_SHARED, serve minor faults with UFFDIO_CONTINUE so clones share pages CoW" pattern is already demonstrated against Firecracker by CodeSandbox (currently on a fork). Cloud Hypervisor exposes a comparable "register in a caller-specified mode, hand the fd to an external manager" design (cloud-hypervisor#8644).
   The one directed question for you
   Would you be open to SharedMemfd (or a sibling backend) accepting a caller-provided, shared, read-only backing fd — i.e. Firecracker maps a fd the caller hands in MAP_SHARED and registers it MODE_MINOR[|WP] — rather than always allocating the memfd per instance? That single decision is what determines whether same-snapshot fan-out can be built on this design or needs a separate backend type. If the answer is yes, items 1/3/4 above are small, and I'm happy to prototype 1 + 2 against snapshot-improvements and report fan-out memory + first-touch latency numbers.