sync truenas/linux-6.12 with release/25.10.5#319
Merged
Conversation
In case consumer provide receive buffer bigger than frame size, misbehaving peer can cause read beyond the frame end. Explicit check should prevent it. (cherry picked from commit fb01b8b)
…by yocalebo) (#317) * TCP connections crossing the NTB interconnect between HA controllers can stall with the peer advertising a zero receive window that never reopens. File copies hang silently, DLM drops its connection for roughly 20 seconds at a time, and long lived websocket sessions see unexplained disconnects. This change fixes the root cause in the ntb_netdev receive path. ntb_netdev posts full MTU sized receive buffers into the transport ring and delivers every received frame to the network stack in that buffer no matter how small the frame actually is. At the default 64KB transport MTU each buffer rounds up to a 128KB allocation, so a 54 byte pure TCP ACK reaches the socket with a truesize near 131KB, a charge of roughly 2400 times the real data size against the socket receive buffer. TCP sizes its advertised window from truesize based accounting. A burst of small control messages fills the receive buffer after a few dozen packets, the advertised window collapses to zero, and the window regrowth paths are gated on incoming packets having a sane length to truesize ratio, which never happens on this device. Bidirectional ssh traffic can then deadlock permanently because application level flow control wedges on both sides while both receive windows are shut. Copy received frames of rx_copybreak bytes or less (default 3646, runtime tunable) into a right sized skb allocated from the page frag cache and recycle the jumbo ring buffer immediately. Small control frames then reach the stack with honest truesize and receive windows stay open. Bulk frames keep the existing zero copy buffer swap. The receive hot path also stops performing a high order atomic allocation for every small frame. Measured on H20 and F60 HA pairs at 64KB MTU using a reproduction harness that drives bidirectional bulk copies, small message chatter and CPU load, comparing builds that differ only by this patch. - truesize per small packet drops from 131328 bytes to 960 bytes - receive buffer overcommit drops from about 1651x to about 12x - TCPWantZeroWindowAdv over a four minute load window drops from 24392 to about 1000 - the unpatched build parks multiple sockets in the TCP persist state within three seconds of load while the patched build shows zero wedged sockets across repeated full runs - a production configured F60 accumulated over 600k wanted zero window events in ten hours of ordinary operation before the fix and near zero after - on the 6.12 stable train the same defect surfaces as receive queue collapse storms of roughly 170k TCPRcvCollapsed per four minute load window, driven by the same truesize inflation this patch eliminates (cherry picked from commit 0d46f76) * ntb_netdev: size reposted RX buffers to the current MTU The rx_handler() recycle paths re-enqueue an skb using ndev->mtu + ETH_HLEN as the buffer length, but change_mtu() cannot drain an skb an in-flight handler is holding. After an MTU increase such a recycled buffer can be smaller than that length and be overflowed by a full-size frame. Post skb_tailroom(nskb), the buffer's true capacity, so the transport's length check rejects any frame that would overflow it. When the buffer is too small for the current MTU, also reallocate a right-sized one before re-enqueuing so it does not reject every full-size frame for its ring slot until it drains; if that allocation fails the smaller buffer is re-posted unchanged and replaced on a later pass. In steady state the buffer is already large enough, so the hot path gains no allocation. Reported in PR review by @ixhamza. (cherry picked from commit 5b4e034) * ntb_netdev: drop frames shorter than an Ethernet header The copybreak path allocates a right-sized skb of len bytes and then eth_type_trans() reads the 14-byte Ethernet header. For a frame with len < ETH_HLEN that reads uninitialized bytes past the copied data and delivers a malformed runt up the stack. len comes from the peer-controlled hdr->len, so a misbehaving peer (including hdr->len == 0) can reach it. Widen the existing len < 0 guard to len < ETH_HLEN so sub-header frames are counted as length errors and their buffer recycled before either delivery path reaches eth_type_trans(). Frames of at least ETH_HLEN bytes are unaffected. (cherry picked from commit b329744) --------- Co-authored-by: caleb <yocalebo@gmail.com> (cherry picked from commit 716a3c7)
ixhamza
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ntb_netdev posts full-MTU receive buffers (ndev->mtu + ETH_HLEN) into
the transport RX ring and delivers them to the stack as-is, so every
received frame carries the truesize of a maximally-sized allocation
regardless of its actual length. At the default 64KB transport MTU
each posted buffer is a 128KB kmalloc, meaning a 54-byte pure TCP ACK
is charged against the receiving socket's rmem at ~131KB - a ~2400x
overcommit.
TCP receive-buffer accounting is driven entirely by truesize: with
default tcp_rmem a socket fills after ~45 packets of any size, the
advertised window collapses to zero (__tcp_select_window() refuses to
advertise less than one MSS), and window regrowth via
tcp_grow_window() is gated on incoming skbs having a sane len/truesize
ratio, which never happens on this device. In practice bidirectional
TCP between HA nodes (e.g. scp over ssh) deadlocks with both
directions at zero window and the persist timer probing forever, and
latency-sensitive protocols (DLM, websockets) see multi-second stalls
whenever bursts of small messages land on one socket.
Copy frames of rx_copybreak bytes or less into a right-sized skb
allocated from the page-frag cache and recycle the jumbo ring buffer
immediately. Small control frames (TCP ACKs, heartbeats, DLM/ssh
chatter) then reach the stack with honest truesize, keeping receive
windows open, and the RX hot path stops doing high-order GFP_ATOMIC
allocations for every small frame. Bulk frames keep the existing
zero-copy buffer swap.