Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign up[RFC] Snapshotting design proposal #1537
Conversation
| - zero or more disk files (depending on how many the guest had). | ||
|
|
||
| This design allows sharing any read-only resources (such as read-only disks or vmm state) between multiple microVMs to improve density. | ||
| When loading a snapshot, instead of loading the full contents from file to memory at resume time, Firecracker will create a `MAP_PRIVATE` mapping of the memory file resulting in runtime on-demand loading of memory pages. Any subsequent memory writes go to a copy-on-write anonymous memory mapping. |
This comment has been minimized.
This comment has been minimized.
sameo
Jan 31, 2020
Contributor
There are security requirements attached to that design decision. A VM created from an existing snapshot could potentially share a lot with the snapshotted VM, including e.g. entropy and crypto material. I think this should be documented and maybe you're planning to do so later on with the actual implementation. Just a heads-up from me.
This comment has been minimized.
This comment has been minimized.
acatangiu
Feb 3, 2020
Author
Contributor
You are right and we do have plans to address those. We will follow-up with detailed docs on specific topics. We're currently interested in getting feedback on the high-level design.
|
|
||
| ### Current favorites | ||
|
|
||
| 1. Extend the existing serde compliant **bincode** serializer to match our requirements. |
This comment has been minimized.
This comment has been minimized.
sameo
Jan 31, 2020
Contributor
bincode is interesting, compact, and works well over the network.
However, I think it fails a few of your (sensible) above requirements: it's not very well specified and is quite bound to the rust language.
This comment has been minimized.
This comment has been minimized.
sandreim
Feb 3, 2020
Contributor
Hi @sameo . I agree that it does not implement a clearly defined spec, but the actual implementation is very simple and easy to understand, extend and improve upon.
We are aware that is currently bound to the rust language, but at this point there are very few usecases defined (updating block device path) that would benefit from the ability to easily edit the snapshot file from outside of Firecracker. Even if we use bincode as the snapshot format, we can support exporting/exporting it as JSON or provide a tool that can handle all future usecases.
| ### We also looked at these other options | ||
|
|
||
| 1. **Serde-bincode**. Use the upstream [`bincode`](https://crates.io/crates/bincode) crate and add minimal versioning support via using a snapshot translator. | ||
| 2. **Serde-cbor**. Use the upstream [`serde-cbor`](https://crates.io/crates/serde_cbor) crate and the [CBOR](https://tools.ietf.org/html/rfc7049) format. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
sandreim
Feb 3, 2020
Contributor
We are targeting very fast serialization/deserialization and minimal snapshot size. CBOR includes a lot of metadata to describe the contents of the data which increase the snapshot size and serialization/deserialization timings.
| 1. **Serde-bincode**. Use the upstream [`bincode`](https://crates.io/crates/bincode) crate and add minimal versioning support via using a snapshot translator. | ||
| 2. **Serde-cbor**. Use the upstream [`serde-cbor`](https://crates.io/crates/serde_cbor) crate and the [CBOR](https://tools.ietf.org/html/rfc7049) format. | ||
| It combines the strong type definition specific to JSON with a binary format for small snapshot sizes. Does not have out-of-the-box support for versioning. Early experimentation indicates potential need for upstream bugfixes. | ||
| 3. **Serde-protobuf**. We haven't invested much in researching protobuf, but it doesn't seem a good choice. It looks complicated to implement in Firecracker as the [`serde-protobuf` crate](https://crates.io/crates/serde-protobuf) is in its early stages. Another option is to directly use [`rust-protobuf`](https://github.com/stepancheg/rust-protobuf). |
This comment has been minimized.
This comment has been minimized.
sameo
Jan 31, 2020
Contributor
The nice thing about protobuf is that it should be a lot more flexible than e.g. bincode. And definitely cross language, large community, etc.
But the serde crate for it is in its very early stages indeed.
This comment has been minimized.
This comment has been minimized.
sandreim
Feb 3, 2020
•
Contributor
Protobuf is indeed an interesting choice, but using it forces us to create an external schema definition for each one of the structs that we want to serialize/deserialize. What we really want here is to decorate structures and write a derive macro to generate version aware serialization/deserialization code.
Signed-off-by: Adrian Catangiu <acatan@amazon.com>
acatangiu commentedJan 21, 2020
Firecracker snapshots
This PR describes the overall snapshotting proposal.
Fixes #1395