Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Snapshotting design proposal #1537

Open
wants to merge 1 commit into
base: master
from

Conversation

@acatangiu
Copy link
Contributor

acatangiu commented Jan 21, 2020

Firecracker snapshots

This PR describes the overall snapshotting proposal.

Fixes #1395

@acatangiu acatangiu self-assigned this Jan 21, 2020
@acatangiu acatangiu force-pushed the acatangiu:snaps-design-rfc branch from 1aa7feb to 9533853 Jan 24, 2020
docs/snapshotting/design.md Outdated Show resolved Hide resolved
@acatangiu acatangiu force-pushed the acatangiu:snaps-design-rfc branch from 9533853 to c23fc9e Jan 30, 2020
docs/snapshotting/design.md Show resolved Hide resolved
docs/snapshotting/design.md Outdated Show resolved Hide resolved
- zero or more disk files (depending on how many the guest had).

This design allows sharing any read-only resources (such as read-only disks or vmm state) between multiple microVMs to improve density.
When loading a snapshot, instead of loading the full contents from file to memory at resume time, Firecracker will create a `MAP_PRIVATE` mapping of the memory file resulting in runtime on-demand loading of memory pages. Any subsequent memory writes go to a copy-on-write anonymous memory mapping.

This comment has been minimized.

Copy link
@sameo

sameo Jan 31, 2020

Contributor

There are security requirements attached to that design decision. A VM created from an existing snapshot could potentially share a lot with the snapshotted VM, including e.g. entropy and crypto material. I think this should be documented and maybe you're planning to do so later on with the actual implementation. Just a heads-up from me.

This comment has been minimized.

Copy link
@acatangiu

acatangiu Feb 3, 2020

Author Contributor

You are right and we do have plans to address those. We will follow-up with detailed docs on specific topics. We're currently interested in getting feedback on the high-level design.

docs/snapshotting/design.md Outdated Show resolved Hide resolved
docs/snapshotting/design.md Outdated Show resolved Hide resolved
docs/snapshotting/design.md Outdated Show resolved Hide resolved

### Current favorites

1. Extend the existing serde compliant **bincode** serializer to match our requirements.

This comment has been minimized.

Copy link
@sameo

sameo Jan 31, 2020

Contributor

bincode is interesting, compact, and works well over the network.

However, I think it fails a few of your (sensible) above requirements: it's not very well specified and is quite bound to the rust language.

This comment has been minimized.

Copy link
@sandreim

sandreim Feb 3, 2020

Contributor

Hi @sameo . I agree that it does not implement a clearly defined spec, but the actual implementation is very simple and easy to understand, extend and improve upon.

We are aware that is currently bound to the rust language, but at this point there are very few usecases defined (updating block device path) that would benefit from the ability to easily edit the snapshot file from outside of Firecracker. Even if we use bincode as the snapshot format, we can support exporting/exporting it as JSON or provide a tool that can handle all future usecases.

### We also looked at these other options

1. **Serde-bincode**. Use the upstream [`bincode`](https://crates.io/crates/bincode) crate and add minimal versioning support via using a snapshot translator.
2. **Serde-cbor**. Use the upstream [`serde-cbor`](https://crates.io/crates/serde_cbor) crate and the [CBOR](https://tools.ietf.org/html/rfc7049) format.

This comment has been minimized.

Copy link
@sameo

sameo Jan 31, 2020

Contributor

Which of your requirements would serde-cbor not meet?

This comment has been minimized.

Copy link
@sandreim

sandreim Feb 3, 2020

Contributor

We are targeting very fast serialization/deserialization and minimal snapshot size. CBOR includes a lot of metadata to describe the contents of the data which increase the snapshot size and serialization/deserialization timings.

1. **Serde-bincode**. Use the upstream [`bincode`](https://crates.io/crates/bincode) crate and add minimal versioning support via using a snapshot translator.
2. **Serde-cbor**. Use the upstream [`serde-cbor`](https://crates.io/crates/serde_cbor) crate and the [CBOR](https://tools.ietf.org/html/rfc7049) format.
It combines the strong type definition specific to JSON with a binary format for small snapshot sizes. Does not have out-of-the-box support for versioning. Early experimentation indicates potential need for upstream bugfixes.
3. **Serde-protobuf**. We haven't invested much in researching protobuf, but it doesn't seem a good choice. It looks complicated to implement in Firecracker as the [`serde-protobuf` crate](https://crates.io/crates/serde-protobuf) is in its early stages. Another option is to directly use [`rust-protobuf`](https://github.com/stepancheg/rust-protobuf).

This comment has been minimized.

Copy link
@sameo

sameo Jan 31, 2020

Contributor

The nice thing about protobuf is that it should be a lot more flexible than e.g. bincode. And definitely cross language, large community, etc.
But the serde crate for it is in its very early stages indeed.

This comment has been minimized.

Copy link
@sandreim

sandreim Feb 3, 2020

Contributor

Protobuf is indeed an interesting choice, but using it forces us to create an external schema definition for each one of the structs that we want to serialize/deserialize. What we really want here is to decorate structures and write a derive macro to generate version aware serialization/deserialization code.

Signed-off-by: Adrian Catangiu <acatan@amazon.com>
@acatangiu acatangiu force-pushed the acatangiu:snaps-design-rfc branch from c23fc9e to f865474 Feb 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.