[Request]: vminit API for block device operations, initially for freeze/thaw.

[jglogan]

Feature or enhancement request details

Adding a guest agent interface call for control operations (typically ioctl()) on block devices can enable some valuable storage features for CZ-based apps:

• Freeze/thaw to ensure clean container commit --live ([Request]: add --live flag to container export container#1400)
• Trim to relinquish freed filesystem blocks to the host ([Request]: Relinquish freed filesystem blocks back to macOS. container#1402).

Let's do the API design in this issue first, before submitting any PR. We don't want individual guest agent calls for each - that would be a mess.

Code of Conduct

• I agree to follow this project's Code of Conduct

[saehejkang]

@jglogan Couple of things

1. Looks like the trim work overlaps with these two issues. Maybe those dupe issues should be closed?

• [Request]: Guest agent API call for returning freed disk blocks to macOS. #542
• [Request]: Storage - Automatically reclaim unused disk space from containers with minimal user interaction #414.

2. Is this issue going to be only focusing on the implementation of freeze/thaw? Or as you mentioned, we want one API that does: freeze, thaw, trim?

3. Any tips on how to get started into understanding the containerization side of things? Here is what I am thinking below:

• Start by looking into Vminitd and understanding how the API works with what exists now.
• Next, I will take a look into how the Ext4 Filesystem works and see how it is being used currently.
• Finally, look into the ContainerizationOS package and understanding how syscall or sysctl works? I feel I need to implement something similar for the ioctl calls (FSTRIM, FSFREEZE).

I want to make sure I gain a solid understanding of things before I even think of coming up with an API design 🚀.

CC: @dcantah in case you had any tips :)

[jglogan]

@saehejkang thanks!

1. I looked for that issue and couldn't find it, so I created a new one. Let's use the earlier [Request]: Storage - Automatically reclaim unused disk space from containers with minimal user interaction #414 and I'll merge any useful informative content from [Request]: Guest agent API call for returning freed disk blocks to macOS. #542 and then close the newer issue.
2. Let's do the general API design for a single call for block device operations here in this issue (and keep bothering @dcantah as he has opinions here). The PR for this issue will cover only implementing the freeze/thaw variant of the API call. [Request]: Storage - Automatically reclaim unused disk space from containers with minimal user interaction #414 will cover trim (which has the potential to be more complicated depending on the mechanism/policy split).
3. Let's look at that step by step.

• Familiarize yourself with the RPC stuff:
  • In the main project, VirtualMachineAgent (the protocol for the client), Vminitd (the implementation over the generated gRPC client), SandboxContext.proto (the protobuf definition, see also the make protos target).
  • In the vminitd subproject, really the main thing you need to look at is Server+GRPC.swift, you'll implement the server side of the call there. Looking at the other code will give you a sense for how to make system calls in Swift.
• You don't need to know anything about ext4fs for this. These ioctls operate at the virtual file system level.
• I don't think you'll need to worry about ContainerizationOS here. This is Linux swift code so you can just call ioctl() (see RunCommand.swift in the vminitd subproject).

The API will definitely need a pathname to the block device, let's call it devicePath for now. I've done the trim function bit in a past life so I can spell out what it will look like in terms of pseudocode; the freeze/thaw case is similar and the device validation code will be shared for whatever operations get performed.

// #define FITRIM _IOWR('X', 121, struct fstrim_range)
FITRIM = UInt32(0xc0185879)

trimBlockDevice(path: FilePath):
    if not path.isAbsolute():
        throw "filesystem trim requires an absolute path"

    finfo = stat()
    rc = lstat(path, finfo)
    if rc != 0:
        throw "filesystem trim stat failed: \(errno)"

    if finfo.mode & MODE_SYMLINK != 0:
        throw"filesystem trim refuses to process a symlink"

    fd = open(r.Path)
    if fd == -1:
        throw "filesystem trim open failed: \(errno)"

    defer fd.close()

    trange = fstrim_range
    trange.start = 0
    trange.len = UInt64.max
    trange.minlen = 0
    rc = ioctl(fd, FITRIM, &trange)
    if rc == -1:
        throw "filesystem trim failed: \(errno)

[saehejkang]

In the vminitd subproject, really the main thing you need to look at is Server+GRPC.swift, you'll implement the server side of the call there. Looking at the other code will give you a sense for how to make system calls in Swift.

@jglogan 😆 My xcode was bugging and the vminitd package was completely missing. I was so confused, as I thought there was some sort of server side call completely missing. I was reviewing your work on #669, when I saw the package! Just as you responded here, I figured out the things you mentioned above (the RPC stuff) and have one clarifying question.

---

I understand that vminitd creates a gRPC server inside the VM listening on a vsock port. So these APIs are created to simply interact with the guest agent (make modifications, etc.). But how does container even use them and is there a good command to follow in order to see how it all comes together?

[jglogan]

Container doesn't know about vminitd.

You need to look for things like this in Containerization: https://github.com/apple/containerization/blob/main/Sources/Containerization/LinuxContainer.swift#L558.

The call site for that create func in container is: https://github.com/apple/container/blob/main/Sources/Services/ContainerSandboxService/Server/SandboxService.swift#L249

[dcantah]

As John said, container is just a consumer of this library, it doesn't care about vminitd. The library has a host-side (macOS) and a guest-side (Linux VM). They communicate over gRPC on vsock port 1024 which you found.

• vminitd (vminitd/Sources/vminitd/) — the guest agent. It boots inside the VM, mounts
  essential filesystems, and starts a gRPC server on yee olde vsock port.
• Vminitd client (Sources/Containerization/Vminitd.swift) — the host-side gRPC client stub that
  wraps all the RPC calls.

The gRPC service is called SandboxContext (defined in Sources/Containerization/SandboxContext/SandboxContext.proto) and exposes a bunch of RPCs (we will add more I promise LOL) for process lifecycle, mounts, networking, sysctl, etc. Essentially everything to setup the system such that it's ready to take workloads (containers).

Following the Container Create + Start Flow

The best command to trace is create → start on LinuxContainer:

1. LinuxContainer.create() (Sources/Containerization/LinuxContainer.swift)

• Creates and boots a VM
• Calls vm.dialAgent() which opens a vsock connection to port 1024 and returns a Vminitd client
• Through that client, sends RPCs to set up the guest:
• standardSetup(): mounts /tmp, /dev/pts, etc.
• mount(): mounts the rootfs and writable overlay
• addressAdd(), up(), routeAddDefault(): configures networking
• configureDNS(), configureHosts(): writes /etc/resolv.conf and /etc/hosts

2. LinuxContainer.start()

• Dials the agent again
• Creates a LinuxProcess which sets up vsock listeners for stdio (stdin/stdout/stderr each get
  their own vsock port)
• Sends CreateProcess RPC: vminitd decodes the OCI spec and creates a ManagedContainer
• Sends StartProcess RPC: vminitd launches the actual process (via vmexec or runc), returns
  the PID.

3. On the vminitd side (vminitd/Sources/vminitd/Server+GRPC.swift)

• createProcess(): decodes the OCI spec, applies alterations (uid/gid resolution,
  PATH/HOME setup), creates a ManagedContainer
• startProcess(): calls ManagedContainer.start() which launches the process

Virtualization.framework

All the virtualization bits right now are backed by interactions to Virtualization.framework. All of this code is mostly shoved in VZVirtualMachineInstance. We made a protocol (VirtualMachineInstance) such that you could kind of plug in any virtualization stack, so long as they support the features we need under the hood (vsock, virtio-block, virtiofs, virtio-net) to do IO. Related to Virtualization.framework, how vminitd even gets in the VM is probably worth mentioning:

vminitd and vmexec binaries are compiled as static Linux binaries (musl libc, cross-compiled via Swift's static Linux SDK; see vminitd/Makefile). They get packaged into an EXT4 disk image that becomes the VM's root filesystem.

The flow:

1. cctl rootfs create (Sources/cctl/RootfsCommand.swift) builds a tar.gz archive containing a minimal rootfs: /sbin/vminitd, /sbin/vmexec, and standard directories (bin, dev, sys, proc, run, tmp, mnt, var).
2. That tar.gz is unpacked into a sparse EXT4 filesystem image via EXT4Unpacker (Sources/Containerization/Image/Unpacker/EXT4Unpacker.swift). The library has its own pure-Swift EXT4 formatter in ContainerizationEXT4.
3. The Makefile orchestrates this with make init (Makefile).

Attaching It to the VM

In Mount.swift, when a mount has runtimeOptions: .virtioblk, it's attached as a VZVirtioBlockDeviceConfiguration backed by a VZDiskImageStorageDeviceAttachment. The init filesystem gets the first block device allocation (/dev/vda) via AttachedFilesystem, which uses a simple character allocator ('a', 'b', 'c'...).

Kernel Command Line

The key piece for asking the guest to run vminitd as its init process is in Kernel.swift, linuxCommandline(initialFilesystem:):

args.append("init=/sbin/vminitd") // run vminitd as PID 1
args.append("ro") // mount root read-only

// for ext4 specifically:
args.append("rootfstype=ext4")
args.append("root=/dev/vda") // the init block device

The default args also include console=hvc0 (so we can get serial console logs) and tsc=reliable.

This gets wired into the boot loader in VZVirtualMachineInstance.swift.