Fix TCP RX softirq wake deadlock#400
Merged
Merged
Conversation
Mask TCP global locks against same-CPU NetRx softirq reentry and let the lock-free ISR wake drain resume legacy Blocked socket waiters. Validated with four back-to-back outbound bssh -> inbound bsshd SSH cycles on one Parallels boot; RX counters stayed balanced with processing-held clear. Co-authored-by: Ryan Breen <ryan@breen.com> Co-authored-by: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
irq_save/irq_restoreso same-CPU NetRx softirq cannot reenter while a thread owns a TCP lock.ThreadState::Blockedsocket waiters, preserving existing syscall blocking semantics without editingsyscall/*.Root Cause
The RX stall was a same-CPU lock reentry deadlock. A userspace TCP syscall path could hold
TCP_CONNECTIONSorTCP_LISTENERS, then a NIC IRQ/NetRx softirq ran on the same CPU and enteredhandle_tcp(), which tried to take the same TCP lock. The interrupted owner could not resume, so NetRx stayed in-progress withNET_RX_PROCESSING_HELD=1,NET_RX_SOFTIRQ_ENTRY_TOTAL = EXIT_TOTAL + 1, andNET_PCI_RX_USED_IDXahead ofNET_PCI_RX_LAST_USED_IDX.Negative control artifacts:
/Users/wrb/Downloads/Ralph/breenix-interrupt-io-roadmap-1780056222/turn94-artifacts/negative-control/Proof
Build gate:
cargo build --release --features testing,external_test_bins --bin qemu-uefipassed clean with zero warning/error lines.Required Parallels proof, same boot, no VM restart:
/Users/wrb/Downloads/Ralph/breenix-interrupt-io-roadmap-1780056222/turn94-artifacts/fixed-proof-3/bssh 10.0.1.210 22 wrb --publickey --exec unamefollowed immediately by inbound host -> Breenixssh ... 'echo AFTER && uname'.Darwin, rc=0.AFTERand Breenixuname, rc=0.Counter evidence from the fixed run:
NET_RX_SOFTIRQ_ENTRY_TOTAL=110,NET_RX_SOFTIRQ_EXIT_TOTAL=110,NET_RX_PROCESSING_HELD=0,NET_PCI_RX_USED_IDX=180,NET_PCI_RX_LAST_USED_IDX=180.NET_RX_SOFTIRQ_ENTRY_TOTAL=237,NET_RX_SOFTIRQ_EXIT_TOTAL=237,NET_RX_PROCESSING_HELD=0,NET_PCI_RX_USED_IDX=374,NET_PCI_RX_LAST_USED_IDX=374.Additional check:
cargo run -p xtask -- boot-stageswas attempted after the proof. It passed 63 stages including the softirq stages, then timed out on the existing x86 ARP gateway marker ([22/252] ARP reply received and gateway MAC resolved). QEMU was killed afterward.