fix: sharded RWMutex to eliminate lock contention in xray server management - #90
Conversation
Add CloseImmediately() function that synchronously closes xray instances instead of relying on the delayed sweeper mechanism. This prevents goroutine and memory leaks when testing high volumes of vless/vmess/trojan proxies. The sweeper's 30-second DrainTimeout caused accumulation of xray instances when tests completed faster than the timeout, leading to resource exhaustion.
The getServer() function was incorrectly resetting DrainedAt to zero, which 'revived' instances that were pending closure. This blocked the sweeper from ever closing them, causing a memory leak. Now returns nil for draining servers, forcing caller to create fresh instances.
…ver/Close - 256 shards with individual sync.RWMutex per shard - getServer: RLock per shard (concurrent readers) - setServer/Close: Lock per shard (no cross-shard contention) - sweeper: per-shard Lock during scan, then tryCloseAndDelete - hashShard() uses FNV-1a for uniform distribution - Update tests to use injectServer/getFromShard/existsInShard helpers - Tests for shard-based CloseAll and drained-entry getServer behavior Fixes proxy-mux hang where 106+ goroutines block on sync.Mutex in getServer during high-volume proxy testing (URLs don't repeat, map miss rate ~100%)
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
sweeper, you can likely replace the shardLockwithRLockwhen scanningshardedServersfor expired entries, since you only read state there and the actual close/delete happens later intryCloseAndDelete, which would reduce contention from the sweeper itself. - In tests (e.g.,
TestTryCloseAndDelete_RevivedEntry), you directly manipulateshardedMuandshardedServersinstead of going through theinjectServer/helper layer; consider adding a small helper for "revive" (resettingDrainedAt) so tests don't need to reach into the sharded internals and stay robust if shard internals change.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `sweeper`, you can likely replace the shard `Lock` with `RLock` when scanning `shardedServers` for expired entries, since you only read state there and the actual close/delete happens later in `tryCloseAndDelete`, which would reduce contention from the sweeper itself.
- In tests (e.g., `TestTryCloseAndDelete_RevivedEntry`), you directly manipulate `shardedMu` and `shardedServers` instead of going through the `injectServer`/helper layer; consider adding a small helper for "revive" (resetting `DrainedAt`) so tests don't need to reach into the sharded internals and stay robust if shard internals change.
## Individual Comments
### Comment 1
<location path="xray/xray.go" line_range="90-93" />
<code_context>
+ sweeperWG sync.WaitGroup
)
+func hashShard(proxyURL string) int {
+ h := fnv.New32a()
+ h.Write([]byte(proxyURL))
+ return int(h.Sum32()) % ShardN
+}
+
</code_context>
<issue_to_address>
**suggestion (performance):** Consider a lighter-weight hash to avoid allocating a new hasher on every shard lookup.
hashShard creates a new FNV hasher on each call, incurring allocations and extra work just to map into 256 buckets. Given the potential call volume from getServer/setServer/tryCloseAndDelete, this may become a hotspot. Consider a simple non-cryptographic string hash (e.g., a small 32-bit rolling hash) or reusing hashers via sync.Pool to cut allocations.
Suggested implementation:
```golang
import (
"sync"
"time"
```
```golang
func hashShard(proxyURL string) int {
// Lightweight non-allocating FNV-1a style hash specialized for strings.
var h uint32 = 2166136261
for i := 0; i < len(proxyURL); i++ {
h ^= uint32(proxyURL[i])
h *= 16777619
}
return int(h % uint32(ShardN))
}
```
</issue_to_address>
### Comment 2
<location path="xray/xray_test.go" line_range="135-138" />
<code_context>
- mu.Lock()
- defer mu.Unlock()
for _, port := range []int{1080, 1081, 1082} {
key := "socks5://127.0.0.1:" + itoa(port)
- if servers[key].DrainedAt.IsZero() {
+ srv := getFromShard(key)
+ if srv == nil || !srv.DrainedAt.IsZero() {
t.Errorf("expected server %s to be draining after CloseAll", key)
}
</code_context>
<issue_to_address>
**issue (testing):** CloseAll test assertion is inverted and will fail when servers are actually draining.
In `TestCloseAll`, `DrainedAt` should be non-zero for draining servers. The current check:
```go
srv := getFromShard(key)
if srv == nil || !srv.DrainedAt.IsZero() {
t.Errorf("expected server %s to be draining after CloseAll", key)
}
```
errors when `DrainedAt` is non-zero. It should instead fail when `srv` is nil or `DrainedAt` is still zero:
```go
if srv == nil || srv.DrainedAt.IsZero() {
t.Errorf("expected server %s to be draining after CloseAll", key)
}
```
so the test matches the intended behavior.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| for _, port := range []int{1080, 1081, 1082} { | ||
| key := "socks5://127.0.0.1:" + itoa(port) | ||
| if servers[key].DrainedAt.IsZero() { | ||
| srv := getFromShard(key) | ||
| if srv == nil || !srv.DrainedAt.IsZero() { |
There was a problem hiding this comment.
issue (testing): CloseAll test assertion is inverted and will fail when servers are actually draining.
In TestCloseAll, DrainedAt should be non-zero for draining servers. The current check:
srv := getFromShard(key)
if srv == nil || !srv.DrainedAt.IsZero() {
t.Errorf("expected server %s to be draining after CloseAll", key)
}errors when DrainedAt is non-zero. It should instead fail when srv is nil or DrainedAt is still zero:
if srv == nil || srv.DrainedAt.IsZero() {
t.Errorf("expected server %s to be draining after CloseAll", key)
}so the test matches the intended behavior.
- hashShard: replace fnv.New32a() allocator with inline FNV-1a loop - sweeper: use RLock instead of Lock when scanning for expired entries - TestSweeperSkipsRevivedEntry: rewrite to match getServer() no-revive behavior - TestCloseAll: fix inverted assertion (!srv.DrainedAt.IsZero() -> srv.DrainedAt.IsZero()) - TestTryCloseAndDelete_RevivedEntry: use reviveServer() helper instead of direct shardedMu access - CloseImmediately: migrate from old mu/servers to shardedMu/shardedServers - Add reviveServer() helper for test injection
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #90 +/- ##
==========================================
+ Coverage 11.85% 13.36% +1.51%
==========================================
Files 27 27
Lines 2194 1594 -600
==========================================
- Hits 260 213 -47
+ Misses 1918 1365 -553
Partials 16 16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Summary
Replace the global mutex protecting the servers map with 256 sharded RWMutexes to eliminate lock contention when high-volume proxy tests call getServer() concurrently.
Root Cause
proxy-mux had ~5000 goroutines blocked on a single global mutex around getServer(), causing single-core CPU saturation and service hangs during load tests with many proxy rotations.
Changes
Commits (5)
a292e98feat: add CloseImmediately for synchronous xray instance shutdowne2e3d3ffix: make Close() synchronously close xray instances24f5368fix tests: update to match synchronous Close() behavior84e8caffix: prevent getServer from reviving draining xray instancesbcd57bafix: sharded RWMutex to eliminate lock contention in getServer/setServer/Close