Drafter is a compute primitive with live migration support.
It enables you to:
- Snapshot, package, and distribute stateful VMs: With an opinionated packaging format and simple developer tools, managing, packaging, and distributing VMs becomes as straightforward as working with containers.
- Run OCI images as VMs: In addition to running almost any Linux distribution (Alpine Linux, Fedora, Debian, Ubuntu etc.), Drafter can also run OCI images as VMs without the overhead of a nested Docker daemon or full CRI implementation. It uses a dynamic disk configuration system, an optional custom Buildroot-based OS to start the OCI image, and a familiar Docker-like networking configuration.
- Easily live migrate VMs between heterogeneous nodes with no downtime: Drafter leverages a custom optimized Firecracker fork and patches to PVM to enable live migration of VMs between heterogeneous nodes, data centers and cloud providers without hardware virtualization support, even across continents. With a customizable hybrid pre- and post-copy strategy, migrations typically take below 100ms within the same data center and around 500ms for Europe ↔ North America migrations over the public internet, depending on the application.
- Hook into suspend and resume lifecycle with agents: Drafter uses a VSock- and panrpc-based agent system to signal to guest applications before a suspend/resume event, allowing them to react accordingly.
- Easily embed VMs inside your applications: Drafter provides a powerful, context-aware Go library for all system components, including a NAT for guest-to-host networking, a forwarder for local port-forwarding/host-to-guest networking, an agent and liveness component for responding to snapshots and suspend/resume events inside the guest, a snapshotter for creating snapshots, a packager for packaging VM images, and a peer for starting and live migrating VMs over the network.
Want to see it in action? See this snippet from our KubeCon talk where we live migrate a Minecraft server between two continents without downtime:
Drafter is available as static binaries on GitHub releases. On Linux, you can install them like so:
for BINARY in drafter-nat drafter-forwarder drafter-snapshotter drafter-peer; do
curl -L -o "/tmp/${BINARY}" "https://github.com/loopholelabs/drafter/releases/latest/download/${BINARY}.linux-$(uname -m)"
sudo install "/tmp/${BINARY}" /usr/local/bin
doneDrafter depends on our custom optimized Firecracker fork for live migration support. Our optional patches to PVM also allow live migrations of VMs between heterogeneous nodes, data centers and cloud providers without hardware virtualization support.
The Firecracker fork is available as static binaries on GitHub releases. On Linux, you can install them like so:
# Without PVM support
for BINARY in firecracker jailer; do
curl -L -o "/tmp/${BINARY}" "https://github.com/loopholelabs/firecracker/releases/download/release-main-live-migration/${BINARY}.linux-$(uname -m)"
sudo install "/tmp/${BINARY}" /usr/local/bin
done
# With PVM support
for BINARY in firecracker jailer; do
curl -L -o "/tmp/${BINARY}" "https://github.com/loopholelabs/firecracker/releases/download/release-main-live-migration-pvm/${BINARY}.linux-$(uname -m)"
sudo install "/tmp/${BINARY}" /usr/local/bin
donePVM installation instructions depend on your operating system; refer to loopholelabs/linux-pvm-ci#installation for more information. Verify that you have a KVM implementation (Intel, AMD or PVM) available by running the following:
$ file /dev/kvm # Should return "/dev/kvm: character special (10/232)"Using Drafter for live migration also requires the NBD kernel module to be loaded, preferably with more devices than the default (12). You can increase the number of available NBD devices by running the following:
$ sudo modprobe nbd nbds_max=4096To create a VM blueprint, you have two options: download a pre-built blueprint or build one from scratch. Blueprints are typically packaged as .tar.zst archives using drafter-packager.
In this tutorial, we'll use Drafter to run the key-value store Valkey (formerly Redis). We'll use two blueprints: the DrafterOS blueprint, a minimal Buildroot-based Linux OS for running OCI images, and the Valkey blueprint, which contains the Valkey OCI image. Note that you can also include Valkey or any other application directly in the base OS blueprint instead of using an OCI image.
Expand section
To download the pre-built blueprints for your architecture, execute the following commands:
$ mkdir -p out
$ curl -Lo out/drafteros-oci.tar.zst "https://github.com/loopholelabs/drafter/releases/latest/download/drafteros-oci-$(uname -m).tar.zst" # Use `drafteros-oci-$(uname -m)_pvm.tar.zst` if you're using PVM, or `drafteros-oci-$(uname -m)_pvm_experimental.tar.zst` if you're using the PVM experimental version
$ curl -Lo out/oci-valkey.tar.zst "https://github.com/loopholelabs/drafter/releases/latest/download/oci-valkey-$(uname -m).tar.zst"Next, use drafter-packager to extract the blueprints:
$ drafter-packager --package-path out/drafteros-oci.tar.zst --extract --devices '[
{
"name": "kernel",
"path": "out/blueprint/vmlinux"
},
{
"name": "disk",
"path": "out/blueprint/rootfs.ext4"
}
]'
$ drafter-packager --package-path out/oci-valkey.tar.zst --extract --devices '[
{
"name": "oci",
"path": "out/blueprint/oci.ext4"
}
]'The output directory should now contain the VM blueprint files:
$ tree out/
out/
├── blueprint
│ ├── oci.ext4
│ ├── rootfs.ext4
│ └── vmlinux
# ...Congratulations! You've downloaded and extracted your first VM blueprint; be sure to check out the packager reference for more information. Next, let's create a snapshot from it!
Expand section
To build the blueprints locally, you can use the included Makefile with the following commands:
# Build the DrafterOS blueprint
$ make depend/os OS_DEFCONFIG=drafteros-oci-firecracker-x86_64_defconfig # Use `drafteros-oci-firecracker-x86_64_pvm_defconfig` if you're using PVM, `drafteros-oci-firecracker-x86_64_pvm_experimental_defconfig` if you're using the PVM experimental version and `drafteros-oci-firecracker-aarch64_defconfig` if you're on `aarch64`
$ make config/kernel # Optional: Configure kernel
$ make save/kernel # Optional: Write back the kernel configuration to the defconfig
$ make config/os # Optional: Configure DrafterOS
$ make save/os # Optional: Write back the DrafterOS configuration to the defconfig
$ make build/os
# Build the Valkey OCI image blueprint
$ sudo make build/oci OCI_IMAGE_URI=docker://valkey/valkey:latest OCI_IMAGE_ARCHITECTURE=amd64 # Or `arm64` if you're on `aarch64`The output directory should now contain the VM blueprint files:
$ tree out/
out/
├── blueprint
│ ├── oci.ext4
│ ├── rootfs.ext4
│ └── vmlinux
# ...You can optionally package the VM blueprint files using drafter-packager for distribution by running the following:
$ drafter-packager --package-path out/drafteros-oci.tar.zst --devices '[
{
"name": "kernel",
"path": "out/blueprint/vmlinux"
},
{
"name": "disk",
"path": "out/blueprint/rootfs.ext4"
}
]'
$ drafter-packager --package-path out/oci-valkey.tar.zst --devices '[
{
"name": "oci",
"path": "out/blueprint/oci.ext4"
}
]'Drafter doesn't concern itself with the actual process of building the underlying VM images aside from this simple build tooling. If you're looking for a more advanced and streamlined process, like streaming conversion and startup of OCI images, a way to replicate/distribute packages or a build service, check out Loophole Labs Architect.
Congratulations! You've built and packaged your first VM blueprint; be sure to check out the packager reference for more information. Next, let's create a snapshot from it!
Now that you have a blueprint available, you can create a VM package from it by using drafter-snapshotter. Drafter heavily relies on the use of snapshots to amortize kernel cold starts and application boot times. To get started, start the Drafter NAT so that the VM(s) can access the network:
$ sudo drafter-nat --host-interface wlp0s20f3 # Replace wlp0s20f3 with the network interface you want to route outgoing traffic from the VMs toIn a new terminal, start the snapshotter:
Using PVM? Make sure to pass a value to
--cpu-template, e.g.T2, so that your snapshots are portable across hosts/different cloud providers with different CPU models!
$ sudo drafter-snapshotter --netns ark0 --devices '[
{
"name": "state",
"output": "out/package/state.bin"
},
{
"name": "memory",
"output": "out/package/memory.bin"
},
{
"name": "kernel",
"input": "out/blueprint/vmlinux",
"output": "out/package/vmlinux"
},
{
"name": "disk",
"input": "out/blueprint/rootfs.ext4",
"output": "out/package/rootfs.ext4"
},
{
"name": "config",
"output": "out/package/config.json"
},
{
"name": "oci",
"input": "out/blueprint/oci.ext4",
"output": "out/package/oci.ext4"
}
]'The output directory should now contain the VM package files with the snapshot:
$ tree out/
out/
# ...
├── package
│ ├── config.json
│ ├── memory.bin
│ ├── oci.ext4
│ ├── rootfs.ext4
│ ├── state.bin
│ └── vmlinux
# ...You can optionally package the VM package files using drafter-packager for distribution by running the following:
$ drafter-packager --package-path out/valkey.tar.zst --devices '[
{
"name": "state",
"path": "out/package/state.bin"
},
{
"name": "memory",
"path": "out/package/memory.bin"
},
{
"name": "kernel",
"path": "out/package/vmlinux"
},
{
"name": "disk",
"path": "out/package/rootfs.ext4"
},
{
"name": "config",
"path": "out/package/config.json"
},
{
"name": "oci",
"path": "out/blueprint/oci.ext4"
}
]'Note that unless you're using PVM (see installation and Live Migrating a VM Instance Across Processes and Hosts for more information), these snapshots are not generally portable across different hosts, even if you use CPU templates. If you're looking for a way to automate snapshot compatibility checks, host-specific builds or a snapshot creation service, check out Loophole Labs Architect.
Cheers! You've created your first VM package; be sure to check out the snapshotter reference for more information. Next, let's find out how we can start it!
Now that we have a VM package, you can start it locally with drafter-peer by running the following:
$ sudo drafter-peer --netns ark0 --devices '[
{
"name": "state",
"base": "out/package/state.bin"
},
{
"name": "memory",
"base": "out/package/memory.bin"
},
{
"name": "kernel",
"base": "out/package/vmlinux"
},
{
"name": "disk",
"base": "out/package/rootfs.ext4"
},
{
"name": "config",
"base": "out/package/config.json"
},
{
"name": "oci",
"base": "out/blueprint/oci.ext4"
}
]'You should see the Valkey logs and a login prompt appear:
# ...
[ 0.902719] crun[383]: 1:M 22 Jun 2024 07:35:10.407 * Server initialized
[ 0.903232] crun[383]: 1:M 22 Jun 2024 07:35:10.408 * Ready to accept connections tcp
Welcome to Loophole Labs DrafterOS
drafterhost login:
By default, input is disabled. To enable it, pass the --enable-input flag to drafter-peer. To stop the VM, press CTRL-C or send the SIGINT signal to the drafter-peer process.
Well done! You've successfully started your VM package; be sure to check out the peer reference for more information. Next, let's find out how you can access it!
See Does Drafter Support IPv6? for more information on the current state of IPv6 support
To access the Valkey instance running inside the VM, you can forward its port to the host using drafter-forwarder. For example, to forward Valkey's TCP port 6379 to port 3333 on your local system (similar to Docker's -p 6379:3333/tcp flag), run the following command:
$ sudo drafter-forwarder --port-forwards '[
{
"netns": "ark0",
"internalPort": "6379",
"protocol": "tcp",
"externalAddr": "127.0.0.1:3333"
}
]'In a new terminal, connect to Valkey with:
$ valkey-cli -u redis://127.0.0.1:3333/0Valkey is now accessible on your local system. To also expose it to your network on port 3333, run the following command (replacing 192.168.12.31 with your IPv4 address):
$ sudo drafter-forwarder --port-forwards '[
{
"netns": "ark0",
"internalPort": "6379",
"protocol": "tcp",
"externalAddr": "127.0.0.1:3333"
},
{
"netns": "ark0",
"internalPort": "6379",
"protocol": "tcp",
"externalAddr": "192.168.12.31:3333"
}
]'Remember to open port 3333/tcp on your firewall to make it reachable from your network. To access the port from both your local system (e.g. for a reverse proxy setup) and the network, you must forward it to both 127.0.0.1:3333 and 192.168.12.31:3333. If you intend on using the port-forwarder during live migrations, keep in mind that connections will break after moving to the new host; if you're interested in keeping the connections alive, see How Can I Keep My Network Connections Alive While Live Migrating?.
Enjoy your Valkey service! Feel free to play around with it using your preferred tooling and check out the forwarder reference for more information. Next, let's find out how we can live migrate it between hosts!
Expand section
To create a single VM instance from a VM package where all changes get written back to the VM package, run the following:
$ sudo drafter-peer --netns ark0 --raddr '' --laddr '' --devices '[
{
"name": "state",
"base": "out/package/state.bin",
"overlay": "out/overlay/state.bin",
"state": "out/state/state.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "memory",
"base": "out/package/memory.bin",
"overlay": "out/overlay/memory.bin",
"state": "out/state/memory.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "kernel",
"base": "out/package/vmlinux",
"overlay": "out/overlay/vmlinux",
"state": "out/state/vmlinux",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "disk",
"base": "out/package/rootfs.ext4",
"overlay": "out/overlay/rootfs.ext4",
"state": "out/state/rootfs.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "config",
"base": "out/package/config.json",
"overlay": "out/overlay/config.json",
"state": "out/state/config.json",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "oci",
"base": "out/package/oci.ext4",
"overlay": "out/overlay/oci.ext4",
"state": "out/state/oci.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
}
]'🚀 That's it! You've successfully created, snapshotted and started an instance of your first service with Drafter. We can't wait to see what you're going to build next! Be sure to take a look at the reference, examples and frequently asked questions for more information.
Expand section
drafter-peer supports the use of a copy-on-write mechanism to allow creating two independent VM instances from the same VM package. To use it, run each VM in its own network namespace and define a different overlay and state file for each VM instance. Start the first VM like so:
$ sudo drafter-peer --netns ark0 --raddr '' --laddr '' --devices '[
{
"name": "state",
"base": "out/package/state.bin",
"overlay": "out/instance-0/overlay/state.bin",
"state": "out/instance-0/state/state.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "memory",
"base": "out/package/memory.bin",
"overlay": "out/instance-0/overlay/memory.bin",
"state": "out/instance-0/state/memory.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "kernel",
"base": "out/package/vmlinux",
"overlay": "out/instance-0/overlay/vmlinux",
"state": "out/instance-0/state/vmlinux",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "disk",
"base": "out/package/rootfs.ext4",
"overlay": "out/instance-0/overlay/rootfs.ext4",
"state": "out/instance-0/state/rootfs.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "config",
"base": "out/package/config.json",
"overlay": "out/instance-0/overlay/config.json",
"state": "out/instance-0/state/config.json",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "oci",
"base": "out/package/oci.ext4",
"overlay": "out/instance-0/overlay/oci.ext4",
"state": "out/instance-0/state/oci.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
}
]'Then start the second, independent VM from the same VM package like so:
$ sudo drafter-peer --netns ark1 --raddr '' --laddr '' --devices '[
{
"name": "state",
"base": "out/package/state.bin",
"overlay": "out/instance-1/overlay/state.bin",
"state": "out/instance-1/state/state.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "memory",
"base": "out/package/memory.bin",
"overlay": "out/instance-1/overlay/memory.bin",
"state": "out/instance-1/state/memory.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "kernel",
"base": "out/package/vmlinux",
"overlay": "out/instance-1/overlay/vmlinux",
"state": "out/instance-1/state/vmlinux",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "disk",
"base": "out/package/rootfs.ext4",
"overlay": "out/instance-1/overlay/rootfs.ext4",
"state": "out/instance-1/state/rootfs.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "config",
"base": "out/package/config.json",
"overlay": "out/instance-1/overlay/config.json",
"state": "out/instance-1/state/config.json",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "oci",
"base": "out/package/oci.ext4",
"overlay": "out/instance-1/overlay/oci.ext4",
"state": "out/instance-1/state/oci.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
}
]'You should now have two independent VM instances started from a single VM package. To access both instances on your local system, you can forward each instance's ports to a matching port on the host; to make the first instance's Valkey available on port 3333 on the host, and the second instance's Valkey available on port 4444 on the host, run:
$ sudo drafter-forwarder --port-forwards '[
{
"netns": "ark0",
"internalPort": "6379",
"protocol": "tcp",
"externalAddr": "127.0.0.1:3333"
},
{
"netns": "ark1",
"internalPort": "6379",
"protocol": "tcp",
"externalAddr": "127.0.0.1:4444"
}
]'In a new terminal, you can now connect to the two independent Valkey instances with:
# For the first instance
$ valkey-cli -u redis://127.0.0.1:3333/0
# For the second instance
$ valkey-cli -u redis://127.0.0.1:4444/0🚀 That's it! You've successfully created, snapshotted and started multiple independent instances of your first service with Drafter. We can't wait to see what you're going to build next! Be sure to take a look at the reference, examples and frequently asked questions for more information.
Expand section
To live migrate a VM instance, first start a new VM instance and set its listen address using the --laddr option. If you want the VM to be migratable over the network instead of just the local system, use --laddr ':1337':
$ sudo drafter-peer --netns ark0 --raddr '' --laddr 'localhost:1337' --devices '[
{
"name": "state",
"base": "out/package/state.bin",
"overlay": "out/instance-0/overlay/state.bin",
"state": "out/instance-0/state/state.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "memory",
"base": "out/package/memory.bin",
"overlay": "out/instance-0/overlay/memory.bin",
"state": "out/instance-0/state/memory.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "kernel",
"base": "out/package/vmlinux",
"overlay": "out/instance-0/overlay/vmlinux",
"state": "out/instance-0/state/vmlinux",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "disk",
"base": "out/package/rootfs.ext4",
"overlay": "out/instance-0/overlay/rootfs.ext4",
"state": "out/instance-0/state/rootfs.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "config",
"base": "out/package/config.json",
"overlay": "out/instance-0/overlay/config.json",
"state": "out/instance-0/state/config.json",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "oci",
"base": "out/package/oci.ext4",
"overlay": "out/instance-0/overlay/oci.ext4",
"state": "out/instance-0/state/oci.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
}
]'Once the instance has started, you should see output like this:
2024/06/24 13:43:11 Resumed VM in 139.720884ms on out/vms/firecracker/VKaFJFNJwPwM6QRoVwQV3i/root
2024/06/24 13:43:11 Serving on 127.0.0.1:1337This instance is now the first peer in a migration chain. Now that the VM instance is migratable, start a second peer (the second peer in the migration chain) and set its remote address to that of the first instance using the --raddr option. This can be done either on the local system (to migrate a VM instance between two processes) or over a network to a second host (to migrate a VM instance between two hosts).
If you use PVM and a CPU template (see installation), this process should work across cloud providers and different CPU models, as long as the CPUs are from the same manufacturer (AMD to AMD and Intel to Intel migrations are supported; AMD to Intel migrations and vice versa will fail). If you're not using PVM, live migration typically only works if the exact same CPU model is used on both the first and second host.
In this example, we'll pass --laddr '' to the second instance, which will not make it migratable again, effectively terminating the migration chain. If you want to make the instance migratable again after the migration, effectively continuing the migration chain, set --laddr to a listen address of your choice. To migrate the VM, run the following command, replacing --raddr 'localhost:1337' with the actual remote address if you're migrating over the network:
$ sudo drafter-peer --netns ark1 --raddr 'localhost:1337' --laddr '' --devices '[
{
"name": "state",
"base": "out/package/state.bin",
"overlay": "out/instance-1/overlay/state.bin",
"state": "out/instance-1/state/state.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "memory",
"base": "out/package/memory.bin",
"overlay": "out/instance-1/overlay/memory.bin",
"state": "out/instance-1/state/memory.bin",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "kernel",
"base": "out/package/vmlinux",
"overlay": "out/instance-1/overlay/vmlinux",
"state": "out/instance-1/state/vmlinux",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "disk",
"base": "out/package/rootfs.ext4",
"overlay": "out/instance-1/overlay/rootfs.ext4",
"state": "out/instance-1/state/rootfs.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "config",
"base": "out/package/config.json",
"overlay": "out/instance-1/overlay/config.json",
"state": "out/instance-1/state/config.json",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
},
{
"name": "oci",
"base": "out/package/oci.ext4",
"overlay": "out/instance-1/overlay/oci.ext4",
"state": "out/instance-1/state/oci.ext4",
"blockSize": 65536,
"expiry": 1000000000,
"maxDirtyBlocks": 200,
"minCycles": 5,
"maxCycles": 20,
"cycleThrottle": 500000000,
"makeMigratable": true,
"shared": false,
"sharedbase": false,
"s3sync": false,
"s3accesskey": "",
"s3secretkey": "",
"s3endpoint": "",
"s3secure": false,
"s3bucket": "",
"s3concurrency": 0
}
]'During the migration, on the first instance, you should see logs like the following right before the first process exits:
# ...
2024/06/24 13:55:16 Migrated 16384 of 16384 initial blocks for local device 4
2024/06/24 13:55:16 Completed migration of local device 4
2024/06/24 13:55:16 Completed all device migrations
2024/06/24 13:55:16 Shutting down
On the destination, after all devices have been migrated, you should see logs like this which indicate that the migration has completed successfully:
# ...
2024/06/24 13:55:16 Resumed VM in 60.026997ms on out/vms/firecracker/39zY39Y9Gs8N4wPMs5WoLe/root
If you're not happy with migration performance, encounter resource contention (e.g. CPU or memory pressure) issues, or want to optimize for something other than minimal downtime, here are configuration options for drafter-peer:
--disable-postcopy-migration: Prevents the instance from resuming on the destination before all data is locally available. This reduces total migration time but increases downtime.--concurrency: Sets concurrent workers for migrations (default 1024). Lower values reduce CPU and memory usage (which corresponds roughly toconcurrency*blockSize- with a 1 MB block size, and 1024 concurrency, memory usage will be 1 GB+) but may increase migration time and downtime, especially in high-latency environments.
You can also disable pre-copy migration. Pre-copy migration minimizes downtime by transferring most data to the destination first, then synchronizing changed data until changes between cycles reach an acceptable level. While this reduces downtime, the extra cycles can increase total migration time. To disable pre-copy migration, modify each device in --devices:
{
"name": "oci",
// ...
"maxDirtyBlocks": 0,
"minCycles": 0,
"maxCycles": 0,
"cycleThrottle": 0
}For bandwidth issues between hosts, consider S3-assisted migrations. This continuously synchronizes bulk data to an S3-compatible store as the instance runs. The destination fetches most data from S3, with only recent changes transferred through the P2P connection. Configure this by extending each device in --devices:
{
"name": "oci",
// ...
"s3sync": true,
"s3accesskey": "myaccesskey",
"s3secretkey": "mysecretkeykey",
"s3endpoint": "s3.us-west-2.amazonaws.com",
"s3secure": true,
"s3bucket": "pojntfx-test-plan-instance",
"s3concurrency": 1024
}Note that S3 configuration is only required when first starting a peer; future peers in a migration chain receive this configuration automatically.
🚀 That's it! You've successfully created, snapshotted, started and live migrated your first service with Drafter. We can't wait to see what you're going to build next!
To make getting started with Drafter as a library easier, take a look at the following examples:
- NAT: Enables guest-to-host networking
- Forwarder: Enables local port-forwarding/host-to-guest networking
- Agent and Liveness: Allow responding to snapshot and suspend/resume events within the guest
- Snapshotter: Creates snapshots/VM packages from blueprints
- Packager: Packages VM instances into distributable packages
- Peer: Live migrates VM instances across the network
- Loophole Labs Silo provides the storage and data migration framework.
- coreos/go-iptables provides the IPTables bindings for port-forwarding.
- pojntfx/panrpc provides the RPC framework for local host ↔ guest communication.
- Font Awesome provides the assets used for the icon and logo.
Bug reports and pull requests are welcome on GitHub at https://github.com/loopholelabs/drafter. For more contribution information check out the contribution guide.
The Drafter project is available as open source under the terms of the GNU Affero General Public License, Version 3.
Everyone interacting in the Drafter project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the CNCF Code of Conduct.
