v0.10.10022026110200

This release hardens the engines and the CI process. CI now runs the gVisor analyzers checklocks, checkconst, and checklinkname, and guest probes that check registers, signal frames, fork, exec, and code rewrites on every backend. The probes and an audit of process-wide state found and fixed per-sandbox memory accounting, trampoline slot races of the Windows systrap engine, a half-patched system call site of the Linux amd64 engine, TCP stalls after packet loss, and signal frame fields that differed from Linux.

Tests

• System call functions of the guest while their pages change: the guest program codewrite has the new check code-text-toggle. One thread makes the pages of syscall.Syscall, syscall.RawSyscall, and syscall.Getpid writable and executable, writes one byte back to itself, and makes them executable only. Four more threads call those functions and verify the result. Linux leaves the code unchanged, so every call must return the process ID.
• CPU state of a new process and a new thread: the guest program forkprobe calls clone with every general purpose register, the vector registers, and the FP control state loaded, and the child stores its registers before it runs other code. A forked process and a thread (CLONE_VM and CLONE_THREAD) must start with all of them, and the parent must keep them. After fork and execve, the new program must start with the Linux defaults of the FP control state (amd64: MXCSR 0x1f80 and x87 control word 0x37f; arm64: FPCR 0). TestE2ESystrapForkState runs it on every backend.
• Signal frame contents: the guest program sigframeprobe verifies the frame that the kernel builds for a signal handler. The handler copies the ucontext and the siginfo and then overwrites every register. The checks compare the copy with the registers of the interrupted thread: the general purpose registers, the program counter, the stack pointer, the signal stack and mask, and the vector and FP record (amd64: the xsave area with its software bytes, magic words, and XSTATE_BV; arm64: fpsimd_context, esr_context, and the end marker). They cover a signal in user code, a store to a read-only page and to an unmapped page (si_code, si_addr, trap number, error code, CR2, fault_address, and ESR), a system call that restarts, and a system call that returns EINTR. A handler that edits the frame must change the general purpose, vector, and FP control registers after sigreturn (X0 stays on arm64), and an edited program counter must skip a faulting instruction. TestE2ESystrapSignalFrame runs it on every backend.
• Register state across events: the guest program regprobe fills every general purpose register, the vector registers, and the FP control registers with distinct values, then lets a system call, a page fault, or a signal happen. The signal arrives in user code, in a blocking call that returns EINTR, in a blocking call that restarts, and in a fault that a SIGSEGV handler resolves. The handler overwrites every register, so only a correct sigreturn restores them. TestE2ESystrapRegisterState runs it on every backend. The package internal/acceptprog/guestcpu holds the assembly for linux/amd64 (RAX to R15, XMM0 to XMM15 and YMM0 to YMM15 when the CPU has AVX, MXCSR, and the x87 control word) and linux/arm64 (X0 to X30, V0 to V31, FPSR, and FPCR).
• Code rewrites from several threads: the guest program codewrite has the new check code-churn-threads. Four threads rewrite a code page between mprotect calls, run it, and replace it by a new mapping at the same address. Two more threads change the permissions and the mapping of the pages beside the code pages. A swapper replaces the page that a runner thread executes by a mapping of one of two memory files with the same code. The code calls getpid, so an engine that patches system call instructions must patch and unpatch them under this load. Each run must return the result of the latest version. TestE2ESystrapCodeWrites runs the check on every backend.
• Flaky cpu.max test: TestE2ECPUMaxEnforcement failed on a hosted windows/arm64 runner and on a loaded hvf host. The Sentry bills the wall time of each run of a thread. The old checks read a busy Go program, and each wake-up of its runtime monitor billed the time that the host needed to run it. One busy loop read 1.06 to 1.17 CPU on the runner and 1.5 CPU on a loaded macOS host. The checks "max gives one CPU", "above demand never throttles", and the loop without system calls now run a static loop with one thread. It reads 1.000 CPU in 30 runs on the runner. The check "small quota" compared the loop counts of a loop at a tenth of a CPU and of a loop that never sleeps. On an Apple CPU, a native loop that runs 10 ms and sleeps 90 ms makes 0.02 of those iterations. A second version compared with a loop of the same duty cycle. On two windows/amd64 runs that loop made about half of the expected progress, and the check failed with shares of 1.9 and 1.833. The cause is probably a change in the speed of the host CPU between two windows. The check now runs a clock loop, which measures its running time with the clock of the guest. It asserts four values. The billed time is 0.07 to 0.13 CPU. At least 0.8 of the periods throttle. The time held back is 0.80 to 0.97 of the wall time. The running time is 0.75 to 1.2 of the billed time. The share is 0.86 to 0.98 on hvf, on windows/amd64, and on windows/arm64. The Sentry code did not change.
• gVisor analyzers in CI: the job build, vet, and audit now runs checklocks, checkconst, and checklinkname on every package for the five targets. A lock annotation violation, a changed assembly offset, or a mismatched //go:linkname fails CI. scripts/gvisorlint.sh runs the same check locally, and docs/gvisorlint.md states how to fix a finding. The first run reported annotation gaps and lock calls that tests skip. It led to the annotations +checklocksacquire, +checklocksrelease, and +checklocksforce on lock helpers, closures, and TryLock calls, and it split the atomic policies of Stack.fib from the rules that routeMu guards.

Bug Fixes

• Close of a Windows named pipe listener of the shim: go-winio 0.6.2 can block win32PipeListener.Close forever. Its listener routine takes the close request while a pending connect ends with an error other than "closed". The routine does not record the close. It waits for a second request, and Accept can return a raw ERROR_OPERATION_ABORTED. TestWindowsLogPipeRejectsCrossInstanceCollision hung for 10 minutes on windows/amd64 in CI. Every pipe listener of the shim (task, stop, and log) now repeats Close until the routine exits. Accept returns net.ErrClosed after Close. Close gives up after 5 seconds and returns an error. The log sink then does not wait for its accept goroutine. The stop server and the shutdown of the shim wait at most 5 seconds after the drain deadline for an accept loop that does not return. TestWindowsPipeListenerCloseRacesPendingConnect uses the real listener.
• Half patch of a system call after a mapping swap on Linux amd64: the systrap engine of Linux patches a mov $nr,%eax; syscall site in three writes. It looked up the frame of the site again for each write. A guest thread could replace the mapping of the page between two writes, for example with mmap and MAP_FIXED of a memory file. The first writes then went into the old frame and the last writes went into the new frame. The old frame kept the invalid byte 0x06 in place of syscall, and the guest died with SIGILL when it ran that frame again. code-churn-threads failed in 15 of 300 runs on a hosted linux/amd64 runner with debug logging. The three writes of a patch and of an unpatch now run in one call of the new method MemoryManager.WithPinnedMappings. A thread that replaces the range waits for the call. The patch also compares the site with the mov and syscall that trapped, and it skips the patch when another thread changed the code.
• Lost reply of Shutdown and ShutdownSandbox: the shim closed the connection right after it wrote the reply. The ttRPC 1.2.7 client cancels its context when it reads end-of-file. If the goroutine of the call did not yet take the queued reply from its stream, the call returned ttrpc: closed for a successful shutdown. The shim waited 20 ms to hide this, and TestShutdownReplyDrainsBeforeServerShutdown failed on loaded linux/arm64 and windows/arm64 runners. A connection that wrote a response now stays open until the peer closes it or one second passes, and the drain waits for it. A client that keeps its connection open after a reply delays the exit of the shim by up to one second. TestShutdownReplySurvivesClientCallLag delays the call goroutine of a real ttRPC client and failed in 7 of 20 runs before the fix.
• uc_stack.ss_flags of a signal frame: the kernel set SS_ONSTACK whenever the handler ran on the alternate signal stack. Linux sets it only when the interrupted code ran on that stack, so a program that checks the flag to detect a nested signal saw a wrong value. The flag now follows Linux in both directions: a handler without SA_ONSTACK that interrupts code on the alternate stack also sees SS_ONSTACK. The guest program sigframeprobe checks it on every backend.
• Threads that continue inside a patched system call on Windows amd64: a thread could start at the return address of a patched syscall, or return from a slot, while another thread patched the page. The patch covered that address with a jmp, and the guest died with SIGILL in syscall.rawSyscallNoError. A guest that changed the protection of its own text caused this, and so did a signal handler that returned to an interrupted call. The engine now keeps every patch away from the addresses where threads continue. A site keeps...

v0.10.10012026110200

This release fixes a regression of v0.10.10002026110200: a signal corrupted the floating point registers of an arm64 guest on Linux and macOS hosts. It also stops the TaskSize search after one probe on a host with a hardware address space size.

Bug Fixes

• arm64 floating point state after a signal: in v0.10.10002026110200 a signal and its sigreturn corrupted the floating point registers of an arm64 guest. On a Linux host, FPSR and FPCR became 0x46508001 and 0x210, so the divide-by-zero trap was on and 1.0/0.0 raised SIGFPE on a CPU that traps. On macOS, the low 64 bits of V0 changed. The signal frame code read the floating point state as a whole fpsimd_context record, but the platforms keep only its body, and hvf kept V0 to V31 first. Every arm64 platform now keeps FPSR, FPCR, and V0 to V31 in this order, and the signal frame code converts the fields one by one. rungv with systrap inside a gvx guest on arm64 also gets the right floating point state in its stub.

v0.10.10002026110200

This release runs rungv with the systrap engine inside a gvx guest on Linux amd64 and arm64, and inside upstream runsc on Linux amd64. The guest kernel now gives the signal frames, the fault information, the mmap errors, and the ptrace behavior that an inner Sentry needs, as Linux does.

Features

• rungv inside a gvx guest and in upstream runsc: rungv run -platform=systrap runs inside a gvx guest on Linux amd64 and arm64, and on macOS arm64 with -platform=hvf. It also runs inside Docker with runsc-kvm and runsc-systrap on Linux amd64. The arm64 case in runsc does not run, because the arm64 signal frame of runsc has no fpsimd_context record. docs/cli.md section 11 lists the flags, the results, and the limits. The guest kernel now meets what the stub of an inner Sentry needs:
  • An unaligned MAP_FIXED mmap above the task size fails with ENOMEM, as in Linux. The size probe of the inner Sentry depends on it.
  • The arm64 signal frame holds the fpsimd_context record, the esr_context record of a fault, and the end marker. sigreturn loads the floating point registers from the frame. Before this release the stub stopped with a wrong magic number, or looped for ever on the missing end marker.
  • The amd64 signal frame of a SIGSEGV or SIGBUS from a page fault holds the trap number 14 and the page fault error code, as in Linux.
  • PTRACE_SETREGS, PTRACE_SETREGSET, and PTRACE_POKEUSER tell the address space when they change the GS base on amd64, as arch_prctl does. The systrap engine then removes its syscall patches that use GS. Before this release a thread that a tracer attached to died in a patched syscall.
  • The host address space size is a search, not a list. It works below the hardware size and on a host that is a Sentry.
  • The stub filter allows any val3 in the futex wake of the sysmsg code.
  • A stub unmaps its address space around a sealed mapping of the host kernel.
  • The syscall thread of a stub on amd64 calls arch_prctl(ARCH_SET_GS) once, so that an upstream runsc host removes its syscall patches from the stub.
  • A fault that a host reports with the error code 0 (amd64) or the ESR value 0 (arm64) gets an inferred access: an instruction fetch when the fault address equals the instruction pointer, a read first, and a write for the second fault at the same instruction and address. Linux never reports 0 for a page fault, so no other host takes this path.
• TestE2ENestedRungvInRungv: runs rungv with systrap in a rungv guest.
• TestE2EPtraceGSBaseKeepsPatchedThreadAlive: attaches to a running thread, sets its GS base with each ptrace request, and verifies that the thread keeps running its patched syscalls.

v0.9.10012026110200

This release fixes the memory limit of a sandbox in a process with several sandboxes and a race of the Windows systrap engine. CI now runs the same tests on windows/arm64 as on windows/amd64, with the race detector on windows/amd64.

Bug Fixes

• Memory limit of a sandbox: Resources.MemoryBytes now counts only the memory of its own sandbox. The limit counted the memory of every sandbox in the process, so an SDK user that ran several sandboxes in one process saw ErrMemoryLimit in a sandbox that stayed far below its limit.
• Registry tests on a host without a hypervisor: TestE2EGateM2BusyboxIsStatic, TestE2EGateM2BusyboxAshInteractive, TestE2EGateM3ArchitectureSelection, TestE2EGateM4TortureUnderDefaultLockdown, and TestE2ERealLinuxBusyboxPythonNodeAcrossBackends named the whpx backend and failed on the windows-11-arm runner, which has no hypervisor. They now skip there when GVX_ALLOW_NO_HYPERVISOR=1.
• Windows systrap: a guest no longer stops with "the stub trapped from unknown slot" when one thread makes a system call while another thread changes the memory of the same code page. The Sentry kept no record of a freed trampoline slot, so it could not decode a trap that a thread published just before the free. A freed slot now keeps its record until a new slot reuses its bytes. The thread of such a trap resumes through the Sentry and not through the freed slot, because a new slot can overwrite the slot bytes at any time.

Tests

• Weekly Windows scope: the Windows jobs of e2e-images.yml run every registry gate, as the Linux jobs do: one step for go test ./e2e without TestE2EK3s*, and one step for the k3s tests. windows-latest runs each gate on systrap and whpx. TestE2EK3sMultiNodeMixedKernels and TestE2EK3sConformance* stay Linux-only: the first needs a native k3s server, and the second runs on linux/amd64 only because of its cost. TestE2EK3sMultiNodeControlPlaneFailover does not run on windows-11-arm: it failed in every run there, because three k3s servers saturate the four cores and the first server loses its leader election when the second joins. It passes on windows-latest.

v0.9.10002026110200

This release makes a gvx node a fuller Linux node for Kubernetes. NetworkPolicy works with kube-router and Calico through IP sets. The wireguard link, the connection tracker, the kernel log, /proc/diskstats, and policy routing follow Linux. Limits without a Linux knob are gone, and host-side values are tunable with --tune. It also adds --host-ca-certs and fixes several bugs of nested and overlay use.

Features

• --host-ca-certs: rungv run --host-ca-certs and Config.HostCACerts make the guest trust the root certificates of the host, for a host behind a proxy that inspects TLS or a host with a private registry. The flag is off by default. rungv reads the roots before the host lockdown: from SSL_CERT_FILE, SSL_CERT_DIR, or the system bundle on Linux; from the System, SystemRootCertificates, and login keychains with the SSL trust settings of the user, admin, and system domains on macOS; and from the ROOT store on Windows. SSL_CERT_FILE and SSL_CERT_DIR add roots on every host OS.
  • At the start, the sandbox appends the roots that an image bundle lacks to each bundle that exists (/etc/ssl/certs/ca-certificates.crt, /etc/ssl/cert.pem, and the RHEL and SUSE paths). An image without a bundle gets /etc/ssl/certs/ca-certificates.crt. The roots that no image bundle holds also become gvx-host-*.crt files in /usr/local/share/ca-certificates and /etc/pki/ca-trust/source/anchors, so update-ca-certificates and update-ca-trust keep them. The writes go to the layer above the image. A path below a mount stays as the mount shows it.
  • Only guest processes trust the roots. A container that dockerd starts does not. docs/docker-engine.md section 6 shows how to give a container or a build the root, and docs/cli.md section 4.10 lists the rules. The sections 6.x of docs/docker-engine.md move to 7.x.
• Calico on k3s: Calico 3.28 in the VXLAN mode with the legacy iptables back end runs on a k3s node (TestE2EK3sCalico). docs/kubernetes.md has the procedure and the limits.
• Policy routing: ip rule (RTM_NEWRULE, RTM_DELRULE, RTM_GETRULE), several routing tables, route metrics, protocols, scopes, and the route types blackhole, unreachable, prohibit, and throw. The guest follows the lookup order of Linux, including fwmark, suppress_prefixlength, and goto. wg-quick with AllowedIPs 0.0.0.0/0 works.
• SO_MARK in the route lookup: The mark of TCP, UDP, raw, and ICMP sockets and of the WireGuard link (fwmark) selects the route.
• iptables for Calico and wg-quick: The match rpfilter, the goto flag of a rule (-g chain), a rule without a target, the target CONNMARK (revisions 1 and 2), and the match connmark. A tracked connection has a mark that conntrack -L and ctnetlink show.
• rp_filter on IPv4 input: The sysctl net.ipv4.conf.*.rp_filter takes effect on IPv4 input (strict and loose), with src_valid_mark.
• Proxy ARP and conf sysctls: A link answers proxy ARP requests. New sysctl files are conf/*/proxy_arp, src_valid_mark, conf/<link>/forwarding, and neigh/*/proxy_delay.
• Queue counts of a new link: RTM_NEWLINK accepts IFLA_NUM_TX_QUEUES and IFLA_NUM_RX_QUEUES.
• net/ipv4/conf/{all,default,<link>}/rp_filter: Calico reads conf/default/rp_filter at start. Before this release the file did not exist, and Calico failed with could not determine default rp_filter setting. The files store any integer and start at 0. A link follows conf/default until a write to its file, and a write to conf/all reaches no link, as in Linux. A new network namespace copies all and default from the root namespace, and later writes stay in each namespace. netstack.Stack.EffectiveRPFilter returns the larger of all and the link value.
• ioctl(SIOCETHTOOL) for offload features: Calico Felix, the ethtool tool, github.com/safchain/ethtool, and vishvananda/ethtool read and change the offload features of a link. Before this release the ioctl returned EOPNOTSUPP, and Felix logged failed to disable checksum offload: operation not supported for vxlan.calico.
  • The commands are ETHTOOL_GFEATURES, ETHTOOL_SFEATURES, ETHTOOL_GSSET_INFO and ETHTOOL_GSTRINGS for ETH_SS_FEATURES, the get and set commands for checksum, scatter-gather, TSO, GSO, and GRO, ETHTOOL_GFLAGS, ETHTOOL_SFLAGS, ETHTOOL_GDRVINFO, ETHTOOL_GLINK, ETHTOOL_GSET, ETHTOOL_GLINKSETTINGS, ETHTOOL_GPERMADDR, and ETHTOOL_GET_TS_INFO. Other commands return EOPNOTSUPP, and a set command without CAP_NET_ADMIN returns EPERM.
  • Each link starts with the features of its Linux driver (loopback, veth, bridge, vxlan, wireguard, and eth0). A set then a get returns the new value. The dependency rules of Linux apply: switching tx-checksum-ip-generic off also turns scatter-gather, TSO, and GSO off, and switching it on brings them back. A feature that the driver fixes returns ETHTOOL_F_UNSUPPORTED or EOPNOTSUPP. The state follows a link into another network namespace and ends with the link. The features do not change how the guest sends packets.
• /sys/fs/bpf and the file system type bpf: the directory /sys/fs/bpf exists in every sysfs mount, as in Linux, and mount -t bpf bpffs /sys/fs/bpf works. Before this release runc failed with mkdirat .../sys/fs/bpf: operation not permitted, a hostPath of type Directory for /sys/fs/bpf failed, and the Calico init container mount-bpffs could not mount. /proc/filesystems lists nodev bpf, and statfs returns BPF_FS_MAGIC. The file system holds directories only, because the guest has no BPF objects to pin. mkdir of an existing name in sysfs now returns EEXIST, not EPERM.
• IP sets, the match set, and the target SET: the ipset tool, kube-router, and Calico create and use IP sets in the guest, and k3s now enforces a NetworkPolicy. Before this release k3s logged Skipping network policy controller start, ipset save failed, and a NetworkPolicy had no effect.
  • The subsystem NFNL_SUBSYS_IPSET of NETLINK_NETFILTER serves the protocol versions 6 and 7: PROTOCOL, CREATE, DESTROY, FLUSH, RENAME, SWAP, LIST, SAVE, ADD, DEL, TEST, HEADER, TYPE, GET_BYNAME, and GET_BYINDEX. ipset restore, batches of elements, and -exist work. An error message carries the request, as in Linux, so the tool decodes it. A listing has no size limit, because the socket sends one message for each read.
  • The set types are hash:ip, hash:net, hash:ip,port, hash:net,port, hash:ip,port,ip, hash:ip,port,net, hash:net,net, bitmap:port, and list:set, with the revisions of Linux 7.0. The options are family inet and inet6, timeout, counters, comment, skbinfo, forceadd, maxelem, hashsize, bucketsize, netmask, bitmask, and nomatch for the net types. A net set tests the longest prefix first, so a nomatch element of a longer prefix wins over a covering net. An element with a timeout expires. The sets belong to the network namespace.
  • Linux limits follow the Linux knobs. The default maxelem is 65536. The file /sys/module/ip_set/parameters/max_sets (default 256) sets the first size of the table of sets of each network namespace that starts after a write, and the table grows by 64 sets, as in Linux.
  • -m set --match-set NAME src,dst (revisions 1 to 4, with !, --return-nomatch, and the counter conditions of the revisions 3 and 4) and -j SET with --add-set, --del-set, --exist, --timeout, and --map-set (revisions 1 to 3) work for IPv4 and IPv6. iptables-save prints the rules back unchanged. A rule keeps a reference to its set, so ipset destroy fails with Set cannot be destroyed: it is in use by a kernel component until the rule ends. iptables finds a set by name through getsockopt(SO_IP_SET).
  • ipset swap exchanges the data behind two names. A rule that names a set sees the new data at once, as in Linux.
• Block statistics from real I/O in /proc/diskstats, /proc/partitions, and /sys/block: the root image and each ext4lite volume are disks of the guest, and their counters count the I/O of the Sentry on the host file. Before this release /proc/diskstats had one disk, vda, that counted the read and write system calls of the processes. A socket, a pipe, or a terminal changed it, a volume had no disk, and cAdvisor in the kubelet could not map the file system of a volume to a disk.
  • The root image is vda (254:0) and it is read-only. Each mounted volume takes the lowest free name: vdb (254:16), vdc (254:32), and so on. A guest without an image has no vda, and its first volume is vda. The disk ends when the mount ends.
  • The files of a volume report the device number of the disk as st_dev. /proc/<pid>/mountinfo and /proc/<pid>/mounts show /dev/vdb as the mount source, and the directories of the root overlay report 254:0. /proc/diskstats, /proc/partitions, /sys/block/<disk>/{dev,size,stat,ro}, /sys/class/block/<disk>, and /sys/dev/block/<major>:<minor> name the same numbers.
  • The counters have the 17 fields of Linux 5.5 and later, in sectors of 512 bytes. One ReadAt or WriteAt call of the volume engine on the image file is one request. One Sync call is a flush, and a Truncate call that shrinks the file is a discard. The times come from the monotonic clock, with the in-flight, I/O, and weighted times of Linux. The merged counts stay 0, because the Sentry has no request queue.
  • The root disk counts the file data that the EROFS driver takes from the image layers. A page of 4 KiB counts once, at its first read or its first guest mapping, as a page cache fill does on Linux, so a second read or a second exec of a file counts nothing. The writes of the guest to its root go to memory, so the sectors written of the root disk stay 0. The metadata of an image is not counted.
  • /proc/<pid>/io is unchanged ...

v0.8.10012026110200

Bug Fixes

• New connections from a pod to an address of the node timed out at random: on a node that forwards packets, a pod that connected to an address of eth0 (the API server at 10.0.2.15:6443, or the cluster IP 10.43.0.1:443 that kube-proxy rewrites to it) got no answer to its SYN for up to 127 seconds. Established connections and connections from the host were not affected. In a k3s node about 1 run in 4 had this fault, and 1 to 4 tests of the [sig-network] Services conformance group failed, with a dial tcp 10.43.0.1:443: i/o timeout or a service is not reachable message.
  • The stack gave a packet for a local address of another NIC to the first NIC that owned the address. Several NICs own such an address, because the loopback NIC holds a temporary copy of every local address that a local route uses, and the stack lists the NICs in random order. The TCP listener then built the SYN-ACK route for that NIC. The route table of a NIC that owns the source address chose the default route through eth0, and the SYN-ACK left the sandbox instead of the bridge. The share of lost SYN-ACKs depended on the number of NICs and on the local connections of the node, so a run had a high or a low rate for its whole time.
  • The stack now delivers such a packet on the NIC where it arrived, as Linux does. It still marks a packet to a broadcast address of the owner as a broadcast, and it still drops a packet from a broadcast source address of the owner.
  • A reply with a local address of another NIC now uses the route table, as in Linux. A route through the NIC of the packet wins. Without such a route, the first matching route decides, so a host with two NICs still answers through the other NIC.

Known limitations

• IPv6 pods: IPv6 has the same random choice of the NIC for a packet to a local address of another NIC, so a SYN-ACK can leave through the wrong NIC. k3s uses IPv4 by default, and the C5 tests use IPv4 only. docs/kubernetes.md, section 4.3, lists the limit.

v0.8.10002026110200

This release lets an unmodified k3s node run in a sandbox (GVXIP-005). It adds the Linux kernel interfaces that the kubelet, containerd, runc, and k3s read and write (phase P1). It adds the service data path of kube-proxy in the iptables mode (phase P2), the wireguard and vxlan links between nodes (phase P3), and the enforcement of cpu.max (phase P4). It adds no flag and no SDK field. docs/kubernetes.md shows how to start a single node and a cluster with two nodes.

Features

• memory.stat at the root cgroup: /sys/fs/cgroup/memory.stat exists and reports all memory of the sandbox, as in Linux. The cAdvisor of the kubelet reads it for the memory of the node, so kubectl top node shows memory. The root still has no memory.current and no memory.max.
• cpu.stat usage of a removed cgroup stays in its parent: usage_usec of a parent counts its live children and the children that it removed. It fell to 0 when a child was removed. Pod cgroups of the kubelet lost the CPU time of finished containers, so kubectl top pod showed 0.
• /dev/kmsg: the character device 1:11 opens for read and write. A non-blocking read returns EAGAIN, and a blocking read waits. lseek with offset 0 returns 0 for SEEK_SET, SEEK_DATA, and SEEK_END. A write goes to the Sentry log and returns its length. The kubelet no longer exits with open /dev/kmsg: no such file or directory. The Sentry has no kernel ring buffer, so a reader never gets a record.
• Sysctls of the kubelet and the container runtime: vm/overcommit_memory accepts 0, 1, and 2 and returns EINVAL for other values. vm/panic_on_oom, kernel/panic, kernel/panic_on_oops, kernel/keys/root_maxkeys, and kernel/keys/root_maxbytes store a value in the range of Linux. net/netfilter/nf_conntrack_max, nf_conntrack_tcp_timeout_established, nf_conntrack_tcp_timeout_close_wait, and /sys/module/nf_conntrack/parameters/hashsize store their values. The files change no behavior of the Sentry.
• ip_unprivileged_port_start: each network namespace has the sysctl. The root namespace of the sandbox is the namespace of the container. It starts at 0, as in a Docker container, and no bind changes there. A namespace that the guest creates starts at 1024, as on Linux. A bind of a TCP or UDP socket of IPv4 or IPv6 to a lower port fails with EACCES unless the caller has CAP_NET_BIND_SERVICE. runc no longer fails every pod sandbox with open /proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory.
• Safe sysctls of Kubernetes: net/ipv4/ip_local_port_range, tcp_syncookies, ping_group_range, ip_local_reserved_ports, tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes, tcp_fin_timeout, and kernel/shm_rmid_forced are writable. The files take the range of Linux, so ping_group_range goes up to 4,294,967,294 and the keepalive and FIN-WAIT-2 times accept 0. The network values belong to the network namespace of the process, so a write in a pod namespace leaves the value of the node. The port range, the keepalive values of new sockets, the FIN-WAIT-2 time, and tcp_syncookies=2 reach the network stack. The stack gets each time as at least one second and the probe count from 1 to 127. ping_group_range and ip_local_reserved_ports only store their values, and tcp_syncookies=0 still sends cookies when the accept queue is full.
• Capacity of the node: with -m, MemTotal in /proc/meminfo and totalram of sysinfo(2) equal the memory limit when it is below the host RAM. The value belongs to each sandbox, so several sandboxes in one process, such as in the SDK, each see their own limit. A tmpfs mount without a size and size=N% still follow the host RAM. With --cpus, the online CPUs, sched_getaffinity, /sys/devices/system/cpu/online, and nproc already showed the whole CPUs of the quota. Without the options nothing changes.
• Generic netlink: socket(AF_NETLINK, SOCK_RAW, NETLINK_GENERIC) works. The controller family nlctrl answers CTRL_CMD_GETFAMILY by name, by ID, and as a dump. It sends the attributes that libnl, libmnl, and github.com/mdlayher/genetlink read. An unknown family returns ENOENT, and an unknown command returns EOPNOTSUPP. A dump is at most 192 KiB. A larger dump fails with ENOBUFS and never ends with a partial list.
• wireguard link (GVXIP-005, phase P3): ip link add wg0 type wireguard, wg set, wg show, and wg-quick work. The link has the MTU 1420. The family wireguard of include/uapi/linux/wireguard.h sets and reads the private key, the listen port (0 picks a free port), the mark, and the peers. A peer has a key, a preshared key, an endpoint, a keepalive, and allowed IP addresses. A read also returns the last handshake time and the byte counters.
  • Both commands need CAP_NET_ADMIN, and the answer holds the private key and the preshared keys, as on Linux. A request that sets the listen port or the mark needs CAP_NET_ADMIN in the user namespace of the namespace that created the link.
  • A set request applies the mark, the port, the removal of every peer, the private key, and the peers in the order of Linux. A listen port that another socket uses fails with EADDRINUSE, and the device keeps its port and its peers. A peer protocol version other than 1 fails with EPFNOSUPPORT.
  • A sandbox has at most 256 wireguard links, and a device has at most 4,096 peers. Both limits fail with ENOMEM.
  • The device is wireguard-go. It sends and receives UDP through the network stack of the namespace where the link was created. The encrypted packets leave through the routes of the guest. A published UDP port (-p 51820:51820/udp) or a keepalive admits the peers.
  • ip link del and the end of the namespace close the device. A netstack link has no down state, so the device owns its UDP port at creation. The mark is stored and reported, but it marks no socket.
• vxlan link (GVXIP-005, phase P3): ip link add vxlan1 type vxlan id 42 dstport 4789 local 10.0.0.1 dev eth0 nolearning works, and so does the call of github.com/vishvananda/netlink that flannel makes. The link is an Ethernet device. Its MTU is the MTU of the underlay link minus 50 (70 for IPv6). That is 1450 on eth0. The MTU stays between 68 and 65,485. ip -d link show and RTM_GETLINK report the kind and the attributes in IFLA_INFO_DATA, so flannel can compare its configuration with the link.
  • A request reads the VNI, the destination port (8472 when absent), local and group (IPv4 or IPv6), and the underlay link. It also reads learning (on when absent), TTL, TOS, DF, ageing, limit, the source port range, GBP, and the UDP checksum flags. The link reports each of them.
  • A VNI above 16,777,215 fails with ERANGE. A second link with the same VNI, address family, and port fails with EEXIST. Links with the same family and port and different VNIs share one UDP endpoint on the wildcard address.
  • A multicast group, external (flow-based), and GPE fail with EOPNOTSUPP. So does a request that changes an existing link with IFLA_INFO_DATA, such as ip link set vxlan1 type vxlan. The same holds for a wireguard link.
  • The link sends each frame as a VXLAN datagram (RFC 7348) through the routes of the network stack that created it. The outer packets leave through eth0 and its NAT, or through wg0. An outer route that leaves through a vxlan link drops the packet.
  • The link receives the datagrams on a UDP endpoint of that stack. It gives each frame to the link with the VNI of the datagram. The outer source port comes from a hash of the flow, inside the source port range, as in Linux. The UDP checksum of an IPv4 outer packet is off unless udpcsum is set.
  • A frame goes to the remotes of the FDB entry for its destination address, or to the all-zeros entry. With learning on, the link learns the source address of each received frame. A learned entry ends after the ageing time, and a frame that would move a static entry is dropped.
  • One FDB holds at most 65,536 entries, or the link limit when that is smaller. A new entry beyond the limit fails with ENOSPC. An entry has at most 1,024 remotes.
  • The link binds its port at creation, because a netstack link has no down state. The link stores and reports ttl inherit, df inherit, label, udp6zerocsumrx, remcsumtx, remcsumrx, remcsum_nopartial, proxy, l2miss, l3miss, and rsc without enforcing them. It does not send to a multicast group.
• Neighbor and FDB netlink (GVXIP-005, phase P3): RTM_NEWNEIGH, RTM_DELNEIGH, and RTM_GETNEIGH work for IPv4 and IPv6 and for the family AF_BRIDGE. ip neigh replace 10.244.2.0 lladdr 02:42:ac:11:00:02 dev flannel.1 nud permanent and bridge fdb append 00:00:00:00:00:00 dst 10.0.0.2 dev vxlan1 work, and so do NeighSet, NeighAppend, NeighDel, and NeighList of github.com/vishvananda/netlink.
  • A permanent or noarp entry becomes a static neighbor of the stack. A dump lists the entries of every link, with the states of the neighbor cache as NUD_* values. A dynamic state in a request fails with EOPNOTSUPP, and so does a link without address resolution, such as lo and wg0. NLM_F_EXCL, NLM_F_REPLACE, and NLM_F_CREATE follow Linux.
  • A link has at most 65,536 static neighbors for each IP family. One more fails with ENOBUFS, and a change of an existing entry still works.
  • An FDB request needs a vxlan link and the state NUD_PERMANENT or NUD_REACHABLE. It reads NDA_LLADDR, NDA_DST, NDA_PORT, NDA_VNI, and NDA_IFINDEX. NLM_F_APPEND adds a remote to the all-zeros address and to a multicast address, and NLM_F_REPLACE changes the remote of a unicast address, as vxlan_fdb_update does.
  • A delete with a remote removes that remote, and the las...

v0.7.10002026110200

This release adds Docker and ext4lite features and fixes HotSpot 21, membarrier, and brk on the Windows systrap engine. It also changes the format of .gvx_meta, so an older release cannot open a workspace that this release wrote. Read the upgrade note under Behavior Changes before the upgrade.

Features

• dockerd --userland-proxy=false: docker:dind now starts with the option. The guest has /proc/sys/net/ipv4/conf/<link>/route_localnet, which changes the martian filter of that link only, and /proc/sys/net/ipv6/conf/<link>/accept_ra for each link, /proc/sys/net/bridge/bridge-nf-call-*, and the hairpin mode of a bridge port. A connection to 127.0.0.1 and a published port reaches the container in the guest, and the host port still works. The bridge does not pass frames through iptables, so --icc=false does not separate containers of one bridge.
• docker stats: the command shows the memory, the CPU time, and the network counters of a container. The guest now reports IFLA_STATS64 for each link, and memory.stat exists in each memory cgroup. A container that wrote 64 MiB shows 64 MiB on hvf and 101 MiB on Linux systrap, because memory.current counts page cache and mapped pages. BLOCK I/O stays at 0B.
• iptables-legacy-save: the command prints the rules of the tables filter, nat, mangle, and raw. The guest has /proc/net/ip_tables_names and /proc/net/ip6_tables_names.
• memory.max of nested cgroups: docker run -m 32m ends the process with the highest score when the container passes the limit. The exit code is 137, docker inspect shows OOMKilled: true, and memory.events counts oom and oom_kill. The Sentry compares anonymous memory plus tmpfs data with the limit, never ends the init process or a process with oom_score_adj of -1000, and checks every 5 ms or more while a container uses half of its limit, otherwise every 25 ms. memory.oom.group ends every process. A container can pass the limit by what it allocates between two checks. The enforcer stays off under the Windows systrap engine. memory.high, cpu.max, and cpu.weight stay unenforced. See docs/docker-engine.md, section 6.4.
• O_TMPFILE on ext4lite volumes: open with O_TMPFILE creates an unnamed file instead of failing with EOPNOTSUPP. The volume frees the file at the last close, and a crash cannot leak it, because the orphan list of ext4 holds the inode. O_TMPFILE without O_EXCL files accept linkat with AT_EMPTY_PATH and through /proc/self/fd. The link moves the inode out of the orphan list in one transaction. A file with no link from O_TMPFILE|O_EXCL or from unlink still refuses linkat with ENOENT.
• fallocate modes on ext4lite volumes: FALLOC_FL_ZERO_RANGE (with and without FALLOC_FL_KEEP_SIZE), FALLOC_FL_COLLAPSE_RANGE, and FALLOC_FL_INSERT_RANGE work. The checks and the error codes follow Linux: block alignment gives EINVAL, a collapse that reaches the end of the file gives EINVAL, and a size above the limit gives EFBIG. Each call is one journal transaction, and e2fsck -fn accepts the image after it.
• RWF_* flags on ext4lite volumes: preadv2 and pwritev2 accept RWF_NOWAIT, RWF_APPEND, and RWF_NOAPPEND, as well as RWF_HIPRI, RWF_DSYNC, and RWF_SYNC for both calls. RWF_APPEND with RWF_NOAPPEND gives EINVAL, and an unknown flag gives EOPNOTSUPP. RWF_NOWAIT never returns EAGAIN, because the volume has no page cache between a call and the engine.
• Hole punching on Windows hostfs: fallocate with FALLOC_FL_PUNCH_HOLE marks the file sparse and calls FSCTL_SET_ZERO_DATA, so NTFS frees the clusters of the range. Before, the call wrote zeros and freed nothing. A volume without sparse files, such as FAT or exFAT, and a file whose NTFS attribute list is full keep the zero-fill. The guest st_blocks still shows the logical size on Windows.

Bug Fixes

• HotSpot 21 on the Windows arm64 systrap engine: java exited with Could not allocate compressed class space: 1073741824 bytes. HotSpot on arm64 asks for the class space at 38 fixed addresses between 4 GiB and 1 TiB, and the guest had no address there. Guests now get all 38 addresses on windows/arm64 and windows/amd64. A middle window from 0x7fff0000 to 66 GiB holds 9 of them, and a window of 2 GiB holds each of the other 29. The high window now starts at 4 TiB instead of 16 TiB. The memory manager supports several address holes, so the platform reports the ranges between the windows. The kernel places the initial stack and the PEB of every stub between 4 GiB and 1 TiB, and it refuses to move the PEB. About 12% of the new stubs therefore hold part of a window. The engine ends such a stub and starts another one, up to 12 times. A platform that drops a window writes a warning to the log. AddressSanitizer and Wine 64-bit still need whpx. TestE2EGateJVM runs Temurin 21 on every backend: C2, eight threads, the garbage collector, a parallel stream, and membarrier. Temurin 25 started on windows/arm64 without the window. See docs/systrap-windows.md, sections 3.10 and 6.2.
• membarrier on hvf, whpx, and Windows systrap: the global and private expedited commands of membarrier(2) returned EINVAL, and MEMBARRIER_CMD_QUERY reported none of them. The commands now succeed, and the query reports them. A call returns after every guest thread that ran guest code at the call has left the guest or stopped once. Idle threads cost nothing. On Windows (whpx and systrap), the call also orders the Sentry threads with FlushProcessWriteBuffers. On macOS, the call orders guest code only and does not order a Sentry thread that copies data for a system call in progress. On whpx, a running guest task returns from Switch once for each call and the kernel runs it again. hvf and Windows systrap resume the guest without a return. Linux platforms keep the host membarrier.
• fsckLite repair of an unlinked inode that a directory names: a power loss between the write of a directory entry and the write of its inode could leave an entry that names an inode with link count zero. The repair kept the count, and the next lookup failed with EUCLEAN. The repair now gives the inode the rebuilt count. Only a mount that runs fsckLite makes this repair, which a volume without a journal needs after a crash.
• Directory link count on Windows hostfs: stat of a directory reported 2 links on NTFS for every directory. It now reports 2 plus the number of subdirectories, as Linux does. Tools such as find use that count.
• Large truncate on Windows hostfs: a growth above the free disk space failed with ENOSPC, and a size above the NTFS file limit failed with EIO. The growth now leaves a hole, as on Linux, and the size above the limit fails with EFBIG.
• brk heap on the Windows systrap engine (windows/amd64 and windows/arm64): brk stopped after 0 MiB to 111 MiB, at a random place in each process. The engine placed a trampoline island at a random address up to 120 MiB above the code, where the heap grows. The engine now looks below the code first. TestE2EGateETExecBrkGrowth moves the break of an ET_EXEC program by 1 GiB on every backend. An arm64 Go program at 0x10000 has no address below its code and keeps the old limit; the Go runtime never calls brk.
• Windows systrap engine errata (windows/amd64 and windows/arm64): three defects of the engine are fixed.
  • A guest that wrote code through one view of a memory file and ran it through another view kept running the old code. A write through a second view of a frame now drops the patched copies of that frame in every address space. A frame without a patched copy costs no extra fault.
  • On windows/arm64, a fast svc overwrote a live x18 of the guest with a stale value, unless the guest set x18 with mov x18, xN. The svc slot now saves x18 first, so every svc keeps it. A guest whose stack pointer is invalid still gets its system call through the debug path. The patcher no longer rewrites mov x18, xN, which leaves 289 rewritten encodings instead of 320.
  • On windows/arm64, every ic ivau of a guest cost one debug stop. The engine now replaces the instruction with nop, and CTR_EL0 reports DIC and IDC for the guest. Guests that obey these bits, such as V8, skip the instruction.
• Windows systrap engine, writes into running code (windows/amd64 and windows/arm64): two kinds of write into code now have the Linux result.
  • A store into the page of the storing instruction, or into the frame that the thread executes through a second view, ended with SIGSEGV. The engine now completes the store with a single step, and the page runs the new bytes. Self-modifying code and data in a writable and executable code page work. On windows/arm64, a store exclusive (stxr, stlxr, stxp, stlxp) into its own page still ends with SIGSEGV.
  • A write of the Sentry into guest memory left a stale patched copy of the page: the copy-out of read, pread, and recv into a mapped page, write to a file that a guest maps, and process_vm_writev. The memory manager and the memory file now report each such write, and the engine drops the copies of the written frames. The report costs one atomic operation for a write into a frame without a copy, and the other platforms keep an empty hook.

Behavior Changes

• fallocate modes other than 0 reach ext4lite and hostfs: the system call returned ENOTSUP for every mode other than 0 on every file system, so FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE never reached a driver from a guest. The call now forwards the mode to a file description that serves modes, which ext4lite and hostfs do. Other file systems keep ENOTSUP. A mode other than 0 skips the RLIMIT_FSIZE check of the system call, because only a call th...

v0.6.10002026110200

This release updates the Sentry to gVisor release-20261102.0 and introduces a new version format. AGENTS.md, section 5.2, defines the format.

Behavior Changes

• Upstream gVisor release-20261102.0: the Sentry now comes from the go branch commit 1cea747e. That commit is the first merge that contains the release commit 7a675ac6. The guest vDSO reads a new CLOCK_BOOTTIME offset from the parameter page.
• Upstream diff budget: tools/upstream -mode=audit accepts a charged share of up to 2.0% of upstream files. The old limit was 1.0%.
• Version format: the PATCH field of a release tag joins the digit 1, a three-digit release counter, the upstream release date, and the two-digit upstream release suffix. v0.6.10002026110200 is the first release in 0.6 and contains gVisor release-20261102.0.

v0.5.2

This release lets Node.js build TypeScript and JavaScript projects on the Windows amd64 systrap engine. The engine runs when the host has no Windows Hypervisor Platform, or with -platform=systrap.

Bug Fixes

• Node.js on the Windows amd64 systrap engine (issue #24): npm install, tsc, transpileModule(), esbuild, and vite build ended with SIGSEGV (exit code 139), SIGTRAP (exit code 133), or SIGILL (exit code 132) in Node.js 20, 22, and 24. Three engine defects caused this:
  • The repair of a lost FS base ran the faulting %fs instruction again, and Windows could reset the base again first. After eight repairs, the engine reported SIGSEGV. The Sentry now runs the instruction and moves past it, so each repair makes progress.
  • The engine patched false system call and GS sites in the data that V8 keeps beside its code in writable and executable anonymous memory. V8 then read changed pointers. On amd64, such pages now run unpatched. A writable segment of an ELF file stays patched.
  • The scan saw only 15 bytes before a page, and it found a GS instruction inside a lea of Node.js 20. The scan now starts 64 bytes before the page.