Repository navigation
v3.2.0: CGNAT peers, removed peers, handshake limits, safe stop
Fixes for open upstream issues and for bugs fixed in other MicroLink forks: one-way tunnels to peers behind CGNAT (upstream #18, #41), peers the control server removes, endless WireGuard handshake retries, and a tailtiny_stop() that could leave tasks running into freed memory. Fixes found in Csontikka, fugo101, antmanler, jetkvm, cplewes and djorr5 were checked against this code and written for it here. Also the idle-power and peer-update work tozes contributed in #13 to #17.
Behaviour changes to know about
- WireGuard goes over DERP until DISCO confirms a direct path. On a LAN that takes about a second; until then a peer's first packets are relayed. Peers behind CGNAT, which could only get one-way traffic before, now work over DERP.
- A handshake that nobody answers is retried for 90 s, then dropped until there's traffic for that peer again (
[WG_PERIODIC] Handshake to key=... unanswered for 90 s, giving up until there is traffic). It used to be retried every 5 s forever. - Peers the control server removes are removed, also from the NVS peer cache (
Peer removed: <name> (<ip>, not in the peer list)). tailtiny_stop()returns once every task has exited, usually in about 100 ms (it always took 3 s), and waits at most 20 s. If a task is still running then, it logs an error andtailtiny_destroy()leaks the instance rather than free it under that task.- TailTiny no longer answers a CallMeMaybe with one.
Fixes
-
WireGuard: a handshake after a reboot is no longer dropped. The handshake initiation's timestamp (TAI64N) was the time since boot. A WireGuard peer drops an initiation whose timestamp isn't newer than the last one it accepted from that key (replay protection; wireguard-go logs
handshake replay), and after a reboot the time since boot starts again near 0. So every initiation TailTiny sent after a reboot was dropped, and a session only came up once the peer started a handshake itself, or had forgotten the old timestamp.wireguard_tai64n_now()now uses the wall clock once it is set (an RTC, SNTP or the application, from 2020 on), and the time since boot only until then. It never goes backwards within a boot, also when SNTP steps the clock back.tests/host/check_tai64n.pychecks it. -
A control-server update no longer resets peers that haven't changed. Every long-poll MapResponse lists all peers again. For each one already known,
add_peer()reset its direct path, re-added it to wireguard-lwip, sent a CallMeMaybe and a forced DISCO probe, and rewrote the whole NVS peer table. With a dozen peers that was about a second of the WireGuard task's time right after connecting, and a flash write per peer, which stalls both cores. Live traffic waited (on an ESP32-S3, TCP to a peer retransmitted during it) and direct paths fell back to DERP until DISCO found them again. A peer whose VPN address, DISCO key, home DERP region and endpoints are unchanged, and that is already in wireguard-lwip, now only takes its online state and hostname, and still gets a CallMeMaybe.ml_peer_nvs_save()skips the flash write when the entry is unchanged, unless an earlier write failed. And a peer whose VPN address changes is removed from wireguard-lwip and added again, so its allowed IP follows (wireguardif_add_peer() kept the old one). -
WireGuard goes only to a DISCO-confirmed address, or over DERP (upstream #18, #41). A new peer's WireGuard endpoint was its first netmap endpoint, an address nobody had checked, and wireguardif sends to any endpoint it has. For a peer behind carrier-grade NAT, or a phone that had left the WiFi whose address it advertised, every reply went there while the peer's packets came in over DERP: a one-way tunnel. The endpoint now stays empty (DERP) until a direct DISCO PONG sets it, as in Tailscale's magicsock; the netmap endpoints are still the DISCO probe targets. A peer that reaches us over DERP while DISCO trusts no direct path to it gets its replies over DERP too (
<peer> reached us over DERP: ...), the endpoint is also forgotten when a direct path's trust expires without a session, andml_wg_mgr_trigger_handshake()only tries the direct address while DISCO trusts it. -
Peers the control server removes are removed.
PeersRemovedholds NodeIDs (numbers), andPeersChangedPatchis an array of changes that name their peer byNodeID; both were read as nodekey strings, so nothing from them was ever applied: a deleted node kept its peer slot (8 without PSRAM) and its DISCO probes, and a peer's new endpoints, home DERP region, DISCO key (a restarted client) and online state only arrived with the next full netmap.OnlineChangewasn't read either. All three are now applied by NodeID. A fullPeerslist now also drops peers it doesn't name, from the peer table and from the NVS cache (which preloaded them on every boot), and a node that comes back under a new node key replaces its old one instead of leaving a second WireGuard peer with the same /32. -
WireGuard handshake retries stop after 90 s, and outgoing traffic starts one. A peer with no session was sent an initiation every 5 s for as long as it stayed
active, also when it was offline or had dropped us: two X25519 operations each, and the chip was never idle long enough to sleep. Retries now stop after 90 s (wireguard-go'sRekeyAttemptTime). In turn, a packet for a peer with no session asks for a handshake, as in wireguard-go; it was dropped (ERR_CONN) without one, and only the endless retries had hidden that. -
WireGuard: a session that stopped answering is re-handshaken after 15 s. When data goes out and nothing authenticated comes back for 15 s (wireguard-go's new-handshake timer), a new handshake starts. A peer that restarted or moved was sent data on its dead session until the rekey, up to two minutes later.
-
WireGuard: the replay check comes before the endpoint update. A packet that decrypted moved the peer's endpoint to its source and refreshed the timers before the replay window was checked (and keepalives weren't checked at all), so anyone who captured one of a peer's packets could resend it from their own address and redirect our packets for that peer. The handshake-initiation rate limit subtracted the wrong way round and never limited anything.
-
The handshake timestamp follows the control server's clock while the system clock isn't set. Without SNTP or an RTC (
basic_connecthas neither), the timestamp was still the time since boot, so a peer still dropped our initiations after a reboot. The MapResponse'sControlTimenow sets it (for WireGuard only; the system clock is left alone). -
A peer reachable on two addresses no longer gets a handshake every 3 s. WireGuard's endpoint follows where the peer's packets come from, and each heartbeat PONG on the DISCO path forced a new handshake when that was the other address (LAN and a NAT hairpin, say). A PONG now switches the session only for a new DISCO path, or one still on DERP.
-
tailtiny_stop()waits for the tasks to exit, and frees what they held. It slept a fixed 3 s, and a task still in a reconnect backoff (up to 16 s), the wait for DERP (15 s), a DERP region probe or a TLS handshake then ran on into queues and an instance thattailtiny_destroy()had freed.ml_net_switchstops and destroys TailTiny when a rebind fails, a transport doesn't come up or the VPN doesn't connect in time, and so does an application that restarts TailTiny. Each task now clears its bit as it exits and stop waits up to 20 s for all of them (if one is still running, destroy leaks the instance instead of freeing it under it). The WireGuard device with its keys (about 15 KB) and the DERP TLS session and socket are freed, andtailtiny_start()after a stop works (the shutdown bit was never cleared, so every new task exited at once). -
DERP: a dead relay connection is noticed. One that died without a FIN or RST (a NAT that forgot it, a router reboot) looked connected forever while we had nothing to send, so peers that reach us over DERP couldn't. The relay sends a keepalive every 60 s; after 150 s with nothing received, TailTiny reconnects. A frame header split across two TLS records no longer drops the connection.
-
Smaller fixes. DISCO no longer derives its box key with an X25519 for every packet (upstream #31; 30-40 ms each on an ESP32-S3), the key per peer is kept. A CallMeMaybe is no longer answered with one: Tailscale never does, and two TailTiny devices kept one going back and forth. The direct one-shot handshake is tried again when DISCO finds a new path. The Noise handshake's reply is read into a buffer of at least 512 bytes and must be 48 bytes long (1 or 2 body bytes after the HTTP 101 overflowed it). The MapResponse JSON is found after its 4-byte length prefix, not at the first
{, which a length byte could be.
New options
CONFIG_ML_TASK_DERP_CORE: the core for the DERP task (0 or 1, default 0, so nothing changes unless set). The DERP relay's TLS handshake took about 3 s of CPU on an ESP32-S3, at priority 5 on core 0, and a lower-priority task there can't run during it: in one product whose UI loop runs on core 0 at priority 1, the screen froze for 2.6 to 2.9 s at every DERP connect. With the option set to 1 the longest stall around the connect was 257 ms, and the handshake took 3.0 to 3.8 s in the background. This moves the contention to core 1, where lower-priority tasks wait instead. Single-core targets always use core 0.
Power
-
Follow-ups to the coord and net_io waits. A STUN socket the coord task creates wakes net_io, so its first reply isn't held for up to 1 s until net_io's
select()times out. The WireGuard task's wait is rounded up to whole ticks: rounded down, it often woke a tick before its job was due and then waited again. The init warning for missing eventfds names both tasks when both lack one. -
The coord and net_io tasks wait for work too, and no task wakes every 100 ms. With the DERP and WireGuard tasks waiting since 3.1.0, the coord task still checked the control connection for 50 ms and then slept 50 ms, and the net_io task's
select()timed out every 50 ms: about 30 wake-ups a second between them, so the chip seldom stayed idle long enough to sleep. The coord task's long-poll now waits in oneselect()on the control connection and an eventfd, which STUN replies, commands, a DERP region change andtailtiny_stop()write. The net_io task'sselect()includes its own eventfd, whichtailtiny_rebind()andtailtiny_stop()write. Both wait up to 1 s, enough for the H2 PING every 5 s and the STUN retries every 2 s. The DERP task waits up to 1 s while connected (still 100 ms while reconnecting), and the WireGuard task until its next timer is due (every 400 ms, unchanged, since that's where a handshake an outgoing packet asks for is sent) instead of every 100 ms. On an ESP32-S3 idle on the tailnet, time in light sleep went from about 13% to about 33%. Each task falls back to its old polling on its own if its eventfd can't be created, withCONFIG_VFS_SUPPORT_SELECToff, and on a cellular AT socket.tests/host/check_idle_wait.pychecks the new wiring.
How it was checked
- All five examples build with ESP-IDF v5.3.2 with no warnings (
basic_connectfor esp32c3 and esp32s3, also withCONFIG_ML_TASK_DERP_CORE=1and withCONFIG_VFS_SUPPORT_SELECT=n; the others for esp32s3). The library is about 5 KB larger. - Host tests pass:
run_tests.py,check_wg_locking.py,check_noise_aead.py,check_idle_wait.pyandcheck_tai64n.py, which now also checks the ControlTime fallback. - On an ESP32-C3 without PSRAM, v3.1.0 plus #13-#17 against this release with the same test firmware: three boots each with device-started traffic (149 of 150 packets arrived in every boot of both), 10 minutes idle with light sleep (69.5% asleep before, 75.3% now), echo after idle and after a rebind (30 of 30 in both), and
tailtiny_stop(): 3018 ms before, 115 ms now, with all tasks gone. - Upstream CamM2325#18 on the same board, with a test patch that drops all direct UDP both ways (so only DERP works) and the PC sending first: before, 0 of 40 echoes came back (the PC re-handshook 8 times, every reply went to the dropped direct path); now 40 of 40, through DERP.
- Not run on hardware: removals and patches from the control server, the 90 s retry limit, the 15 s re-handshake and the DERP receive timeout need a node deleted from the tailnet, a peer going away or a dead relay connection.
Full history: CHANGELOG.md