Skip to content

lib: make restarts hitless by adopting the kernel WireGuard dataplane - #14

Merged
pbardea merged 3 commits into
mainfrom
graceful-restart-adopt-wireguard
Jul 31, 2026
Merged

lib: make restarts hitless by adopting the kernel WireGuard dataplane#14
pbardea merged 3 commits into
mainfrom
graceful-restart-adopt-wireguard

Conversation

@pbardea

@pbardea pbardea commented Jul 30, 2026

Copy link
Copy Markdown

Makes a vprox restart (i.e. any deploy) hitless for existing WireGuard peers. Previously, shutdown deleted the vproxN interface and startup force-recreated it empty, so every registered VM silently lost its tunnel: its config still looked healthy but the server no longer knew its pubkey, and packets were dropped until GitHub's 10-minute lost-communication timer killed the CI job. The kernel dataplane never needed to be touched; it keeps forwarding on its own and already stores everything needed to rebuild the process's memory.

Three changes, all required together:

  1. Start() in lib/server_manager.go no longer defers CleanupWireguard/CleanupIptables, so the interface and iptables rules survive shutdown (the cleanup functions remain for manual decommissioning; StartIptables uses AppendUnique throughout, so re-running it over surviving rules is a no-op).
  2. StartWireguard() adopts an existing vproxN interface instead of delete+recreate, falling back to recreation only when the device's address doesn't match WgCidr. The finishing steps (AddrReplace, LinkSetUp, ConfigureDevice without ReplacePeers) are non-destructive to existing peers.
  3. A new RestorePeersFromKernel() rebuilds peerIPs, ipAllocator, and newPeers from the kernel device's peer dump. This is what makes 1+2 safe: without it, a fresh allocator would hand a new VM an IP an existing peer already owns, and the kernel would silently move that IP's routing to the new peer (AllowedIPs are exclusive), blackholing the old one. Restored peers get the usual connect grace period so the idle reaper doesn't instantly reap peers whose last handshake predates the restart.

The peers-to-state computation is a pure function so it's testable without a kernel WireGuard device:

func restorePeerState(peers []wgtypes.Peer, alloc *IpAllocator) restoredPeers {
	result := restoredPeers{peerIPs: make(map[wgtypes.Key]netip.Addr)}
	for _, peer := range peers {
		addr, ok := peerAssignedIp(peer) // first AllowedIP, must be an IPv4 /32
		if !ok || !alloc.Claim(addr) {
			result.invalid = append(result.invalid, peer.PublicKey)
			continue
		}
		result.peerIPs[peer.PublicKey] = addr
	}
	return result
}

IpAllocator.Claim(addr) is the small new allocator API backing this: it marks a specific address as allocated, refusing addresses outside the prefix or already taken. Peers with missing or malformed AllowedIPs are removed from the device during restore to keep state consistent.

Safety notes: the server's private key already persists in /run/vprox/server-key, so an adopted interface keeps handshaking with existing peers. On a host reboot, tmpfs loses the key but the interface is gone too, so both sides reset together like a cold start today. Crashes (SIGKILL/OOM) get the same benefit, since the orphaned interface is adopted on next start. Result: only /connect is unavailable for the ~1s the process is down; existing tunnels never stop forwarding.

Unit tests cover Claim semantics and restorePeerState (valid peers, missing/non-/32/out-of-prefix/duplicate AllowedIPs). go vet, gofmt, and go test ./lib/ pass.


View with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled. (Staging)


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Co-authored-by: Codesmith Staging <codesmith-bot@users.noreply.github.com>
Comment thread lib/server_manager.go
Co-authored-by: Codesmith Staging <codesmith-bot@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b9b5d0d. Configure here.

Comment thread lib/server.go
Co-authored-by: Codesmith Staging <codesmith-bot@users.noreply.github.com>
@pbardea
pbardea merged commit f1b7657 into main Jul 31, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants