rate_limit: fix SNI queue slot accounting and the sweep/close race#13406
Open
moonchen wants to merge 4 commits into
Open
rate_limit: fix SNI queue slot accounting and the sweep/close race#13406moonchen wants to merge 4 commits into
moonchen wants to merge 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes correctness and concurrency issues in the experimental rate_limit plugin’s SNI queue handling during TLS handshakes, preventing active-slot counter underflow and stabilizing sweep vs. close interactions. Adds gold tests to regression-test the queue, expiry, and reject paths against a TLS listener.
Changes:
- Fix SNI queue slot accounting by reserving before dequeue/resume, detaching expired queued VCs, and only releasing slots for VCs that actually own a slot.
- Synchronize the periodic sweep and
TS_EVENT_VCONN_CLOSEhandling with a shared mutex to prevent sweep/close interleavings corrupting queue / slot / lease state. - Add new AuTest gold tests (plus bash+openssl clients) covering reject, queue, and max_age expiry behaviors.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gold_tests/pluginTest/rate_limit/rate_limit_sni_reject.test.py | Adds an autest covering the no-queue reject path under TLS. |
| tests/gold_tests/pluginTest/rate_limit/rate_limit_sni_reject_client.sh | Bash/openssl client to generate concurrent handshakes to trigger rejects. |
| tests/gold_tests/pluginTest/rate_limit/rate_limit_sni_queue.test.py | Adds a regression autest for queued-VC close vs. slot underflow. |
| tests/gold_tests/pluginTest/rate_limit/rate_limit_sni_queue_client.sh | Deterministic bash/openssl reproducer for the historical underflow scenario. |
| tests/gold_tests/pluginTest/rate_limit/rate_limit_sni_expiry.test.py | Adds a regression autest for the max_age expiry accounting path. |
| tests/gold_tests/pluginTest/rate_limit/rate_limit_sni_expiry_client.sh | Bash/openssl client to hold a slot while a queued VC ages out. |
| plugins/experimental/rate_limit/sni_selector.cc | Fixes sweep logic (reserve→pop→reenable) and detaches expired queued VCs; adds sweep/close synchronization. |
| plugins/experimental/rate_limit/sni_limiter.cc | Fixes close-time accounting (remove-if-queued vs free-slot) and serializes with sweep under the shared mutex. |
| plugins/experimental/rate_limit/limiter.h | Adds remove() to drop still-queued elements so queued closes don’t decrement the active slot counter. |
A queued SNI connection never reserves a slot, but its VCONN_CLOSE released one unconditionally. A queued connection that closed therefore decremented the active-slot counter without a matching increment; it wrapped below zero and the next reserve() aborted the server on TSReleaseAssert(_active <= _limit). Balance the accounting: resume queued connections with reserve-then-pop so a resumed connection owns a real slot; release a slot on close only when the connection is no longer queued (a still-queued one never held one) and drop it from the queue; detach an expired connection the same way the reject path does. Removing a closing connection from the queue also fixes a stale-pointer dereference when a parked queued connection is reset. Add deterministic regressions for the resume and max_age paths.
The SNI active-slot queue sweep (task thread) and the VCONN_CLOSE handler (net thread, inline) touched the shared queue and slot counter with no synchronization. A close racing the sweep could pop an emptied queue and reenable a null VC (crash); free a slot the queued VC never reserved, underflowing _active until the next reserve() aborts on TSReleaseAssert; double-release a lease and delete the selector early; or reclaim a popped VC before the sweep reenabled it. Guard the sweep's reserve->pop->reenable transaction and the close handler's slot/lease accounting under one plugin-global mutex, re-read the user arg under the lock, null-check pop() results, and keep the popped VC pinned across the reenable.
Exercise the sync-reject path against a TLS listener: a holder reserves the one slot and a burst of concurrent handshakes is rejected mid-handshake (TS_EVENT_ERROR) with the allocator freelists disabled. Asserts the reject path is reached and every rejected handshake VC is freed without a memory-safety fault.
moonchen
force-pushed
the
rate-limit-sni-queue-fixes
branch
from
July 19, 2026 17:37
f7e15a5 to
516e40c
Compare
Annotate the TestRun parameters like the surrounding class-based gold tests, and create the holder FIFO inside a fresh mktemp -d directory instead of on an unlinked mktemp -u path, whose creation is not atomic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When an
snirule has alimitand a queue, connections over the limit are parked in a queue at CLIENT_HELLO. A periodic sweep (every 300 ms, on a task thread) resumes queued connections when slots free up and expires ones that have waited longer thanmax_age.Slot-counter underflow. A queued connection never increments the active-slot counter, but closing one decremented it unconditionally. The counter wraps below zero, and the next reservation fails a release assert, aborting the server. The sweep had a related accounting bug: its loop condition was inverted, so it resumed a queued connection precisely when
reserve()failed — waking it without a slot — and stopped, discarding a successful reservation, whenreserve()succeeded.The first commit balances the accounting:
max_ageis detached (user arg cleared, selector lease released) the same way the reject path does, so its close no longer releases a slot it never held.Queued connections now stay parked until a slot actually frees instead of being resumed over the limit — which also makes
max_ageexpiry reachable.Sweep/close synchronization. The sweep and the VCONN_CLOSE handler (net thread, inline) update the queue, the slot counter, and the selector lease in multi-step sequences. Each structure is locked on its own, but the sequences can interleave: a close can empty the queue between the sweep's
size()andpop()(the sweep then reenables a null VC), release a slot the sweep is concurrently granting, double-release the selector lease and delete the selector early, or free a popped connection before the sweep reenables it.The second commit synchronizes the sweep with connection close under one plugin mutex: the sweep's reserve→pop→reenable sequence and the close handler's slot/lease accounting run under the lock, the close handler re-reads the VC user arg under the lock (the expiry path may have detached the VC), and
pop()results are null-checked.Add autests for this plugin. Three tests in
tests/gold_tests/pluginTest/rate_limit/, the first autest directory for rate_limit (bash +openssl s_clientclients against a TLS-enabled ATS withlimit: 1):rate_limit_sni_queue— queue size 1; one connection holds the single slot (a FIFO keeps it open until the test releases it), a second parks in the queue and closes while parked;rate_limit_sni_expiry— same, with the parked connection expired bymax_age: 1;rate_limit_sni_reject— no queue; a burst of connections over the limit is rejected mid-handshake (TSVConnReenableExwithTS_EVENT_ERROR).With an unpatched plugin, the queue and expiry tests fail: traffic_server exits on SIGABRT at
failed assertion '_active <= _limit'(limiter.h), and the underflow is visible in traffic.out (Releasing a slot, active entities == 0followed by... == 4294967295). With these commits they pass and the logs contain no assertion, Fatal, or signal lines. The abort under test is aTSReleaseAssert, so the tests do not depend on ASan or a particular build type. The reject test passes with and without the fixes — it is coverage of the no-queue reject path, which the fixes do not touch.The accounting fix is regression-tested on both the resume and
max_agepaths. The race fix is analytic: the interleavings above follow from the threading model, but a lost race has no deterministic reproduction, so there is no regression test for the second commit.