fix: harden cocoon for sub-1500-MTU / high-vCPU cloud hosts (CH rlimits + CNI MTU/MSS)#135
Merged
Merged
Conversation
Cloud Hypervisor allocates an io_uring ring plus eventfds per virtio-blk queue, and num_queues scales with vCPU, so a high-vCPU multi-disk guest (e.g. an 8-vCPU 10-disk Android VM = 80 queues) exhausts the default RLIMIT_MEMLOCK (~memlock) and RLIMIT_NOFILE (1024), failing disk activation with 'AsyncIo: I/O error' then 'Too many open files' VMM panic. Lift both before fork; the VMM inherits them.
On a host whose egress MTU is below the CNI bridge default 1500 (GCP is 1460), guests silently blackhole large DF packets (TLS/data), so e.g. reaching a CDN with a big cert chain times out while ping/small requests work. Set the bridge mtu to the detected host MTU in the generated conflist, and add a mangle FORWARD TCPMSS --clamp-mss-to-pmtu rule so existing hosts are fixed without bridge rebuild.
Remove unnecessary comments about VMM resource limits.
…recracker The rlimit raise sat in the CH-specific launch path, but Firecracker also forks a VMM with an io_uring block engine (IoEngine=Async) and hits the same RLIMIT_MEMLOCK/NOFILE exhaustion. Both backends go through Backend.LaunchVMProcess, so raise the limits there, right before spec.Cmd.Start().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent robustness fixes surfaced while validating cocoon on a GCP C4A
(arm64) host, each verified on that hardware.
1. Raise RLIMIT_MEMLOCK / RLIMIT_NOFILE before launching the VMM (
hypervisor/cloudhypervisor/start.go)Cloud Hypervisor allocates an io_uring ring plus eventfds per virtio-blk queue, and
num_queuesscales with vCPU. A high-vCPU multi-disk guest — e.g. an 8-vCPU, 10-diskAndroid VM = 80 queues — exhausts the process defaults (
RLIMIT_MEMLOCK~93 MiB,RLIMIT_NOFILEsoft 1024) and fails:(preceded by
failed to create new AsyncIo: I/O erroronce memlock is exhausted).raiseVMMRlimitslifts both before the fork; the VMM inherits them.Verified (arm64 testbed): without the fix an 8-vCPU VM needs a manual
ulimit -l unlimited; ulimit -n 1048576wrapper to boot; with it, a plaincocoon vm run --cpu 8 --memory 8Gboots in ~29 s with 0 io_uring/disk errors.2. Match CNI bridge MTU to the host + clamp forward MSS (
doctor/check.sh)On a host whose egress MTU is below the CNI bridge default 1500 (GCP is 1460),
guests silently blackhole large DF packets: small requests and ping work, but reaching
a CDN with a big TLS cert chain times out (the VM sends 1500-byte segments that the
1460-MTU host egress drops, and the ICMP frag-needed never makes it back). Fix:
mtuin the generated conflist to the detected host egress MTU, andmangle FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmturule so existing hosts are fixed immediately, without a bridge rebuild.
Verified (arm64 testbed): before,
curl https://p0.meituan.netfrom a guest timesout after 8 s (0 bytes); after the clamp it returns in ~0.2 s. The conflist regenerates
with
"mtu": 1460on a GCP host.