feat(rdma): evict stalest idle connection when receive segment is exhausted - #270
Merged
Conversation
…austed Previously an exhausted shared receive segment refused every new connection outright: no eviction path existed (RecvSegment has only Allocate/Release), so a connection storm (instance reconnects, new instances) could starve new connections for the full idle window (30s-10min). Now: each endpoint tracks last_active_us_ (updated by its Serve loop whenever WaitComp yields completions). When Allocate fails, the server wakes the stalest connection idle >= 2s (no completions -> no in-flight request; safe to evict since clients re-dial via the stale-retry path) and polls for the freed lease, up to 32 evictions per admission. Only recently-active connections are protected. New metric dfkv_rdma_segment_evictions_total. Verified on 0064 (64MB segment = 15 conns @ depth 4 x 1MB slots, t16/b1 rounds): without eviction every new connection is refused (16/16); with eviction 15/16 admitted, 1 transient failure per round (concurrent eviction race at the boundary), 0.05% of requests. Active connections are never evicted (all-active admission still refuses).
yuhaijun999
pushed a commit
to yuhaijun999/DingoCache
that referenced
this pull request
Aug 7, 2026
…, mode typo guard Review of v2.6.2 (dingodb#269/dingodb#270) found: 1. SteadyUs() returned steady_clock::count() raw ticks, which are NANOSECONDS on Linux. kEvictIdleMinUs=2,000,000 was therefore 2 ms, not 2 s: under normal traffic nearly every pooled connection counts as idle, so a full segment could evict recently-active connections and trigger reconnection storms. Now duration_cast<microseconds>. 2. Eviction woke a live_eps_ pointer whose owning Serve thread then exited and destroyed the stack endpoint — a concurrent evictor could Wake a freed endpoint (UAF). The evictor now erases the victim from live_eps_ under conn_mu_ before Wake (Serve's own erase becomes a no-op), so exactly one evictor ever touches an endpoint. 3. Total eviction wait is bounded to 5 s (was up to 32 s, exceeding the client's 10 s bootstrap). 4. last_active_us_ == 0 (inserted, not yet stamped by Serve) is never treated as idle. 5. DFKV_RAM_WRITE_MODE accepts only writeback|writearound; anything else warns and falls back to writeback (was silent writeback on typos). Verified on 0064 (64MB segment = 15 conns, t16/b1 rounds): A all-active refusal 26 fails (expected), B-F 1-2 fails each (eviction boundary), 75 evictions, zero crashes/segfaults.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An exhausted shared receive segment refused every new connection outright — no eviction path existed (
RecvSegmenthas onlyAllocate/Release). A connection storm (instance reconnects, new instances coming up) could starve new connections for the whole idle window (30s–10min default).Change
RcEndpoint::last_active_us_: steady-clock timestamp updated by the Serve loop wheneverWaitCompyields completions (both URING and sync loops).Allocatefailure the server wakes the stalest connection idle ≥ 2s (no completions ⇒ no in-flight request; clients re-dial via the existing stale-retry path) and polls for the freed lease, up to 32 evictions per admission. Recently-active connections are never evicted; if every connection is active the admission still refuses (unchanged behavior).dfkv_rdma_segment_evictions_total.Verification (0064, 64MB segment = 15 conns @ depth 4 × 1MB slots, t16/b1 rounds)
Without eviction every new connection is refused (16/16). Request-level impact is 0.05% and covered by client stale-retry.