MongoDB replicaSet loses primary under the platform connector pool load

[nitz2407]

Prerequisites

• I searched existing issues
• I can reproduce this issue

Code of Conduct

• I agree to follow NVSentinel's Code of Conduct

Bug Description

MongoDB replicaSet loses primary under the platform connector pool load, which crash-loops the remediation pipeline (no cordoning / no remediation CRs)

• Discovered: during harness end-to-end run at 5,000 KWOK nodes
• Severity: high — the entire fault → cordon → drain → GPUReset remediation path is unavailable while the platform connector pool is up
• Component(s): mongodb (event store), fault-quarantine, node-drainer
• Area: scale / resource sizing

Summary

At 5,000 emulated (KWOK) nodes, the harness deploys a platform connector
pool of ~1,600 pods to simulate the per-node connector plane, then injects
health events through them. Every connector opens a TLS/X.509 (mTLS)
connection to the shared 3-node MongoDB replicaSet. The X.509 handshake and steady
connection load pin the mongodb container at its CPU limit (1500m = 1.5 cores).
Under that saturation the mongod readiness probe times out, so 2 of the 3
replicas go NotReady and restart every 20–40s. With a majority of members
flapping, the replicaSet has no primary (ReplicaSetNoPrimary).

Because the event store has no primary, the remediation consumers can't connect:

• fault-quarantine enters CrashLoopBackOff (fails on startup datastore ping).
• node-drainer restarts repeatedly.

So the injected fatal events are never turned into cordons or GPUReset CRs.
Observed end state after injecting 6,400 events (~512 fatal): 11 nodes cordoned,
0 GPUReset CRs, 0 reset Jobs — i.e. remediation effectively stopped.

The injection itself succeeds (6,400/6,400 acked); the failure is that the SUT's
event store cannot simultaneously serve ~1,600 connector clients and the
remediation pipeline on the current MongoDB CPU allocation.

Trigger vs. baseline (observed): once the inject/reconcile load stopped, all
three mongod replicas recovered to 2/2 Running and stopped restarting even
though the ~1,600 connectors were still connected. So the collapse is triggered
by the write burst + the concurrent per-shard reconcile query fan-out layered on
top of the connector connection baseline — not the idle connector count by
itself. This is why reducing connector density helps: a smaller baseline leaves
the 1.5-core mongod enough CPU headroom to absorb the burst without missing its
probes.

Environment

Property	Value
Cluster	Azure AKS
Kubernetes	v1.32.5
Emulated nodes	5,000 KWOK GPU nodes (type=kwok, 8× H100 each)
Real worker nodes hosting the pool	~27 nodes carried resident injectors
Platform connector pool	~1,600 connector pods (per-node-pod-limit=50)
Events injected	6,400 (4 events/connector), fatal fraction 0.08 → ~512 fatal
Event store	MongoDB replicaSet rs0, 3 members (mongodb-0/1/2), mTLS MONGODB-X509 auth, requireTLS
MongoDB mongodb container	requests cpu=1, limits cpu=1500m, memory=2Gi
MongoDB metrics sidecar	requests cpu=100m/128Mi, limits cpu=150m/192Mi
NVSentinel	v1.9.0, CSP/janitor = kind (no-op remediation backend)

What was observed (evidence)

MongoDB replicas flapping at their CPU limit

$ kubectl get pods -n nvsentinel -l app.kubernetes.io/name=mongodb -o wide
NAME        READY   STATUS    RESTARTS       AGE
mongodb-0   2/2     Running   16 (11m ago)   14d
mongodb-1   1/2     Running   9  (40s ago)   14d
mongodb-2   1/2     Running   9  (19s ago)   14d

# The non-ready container is mongod itself (metrics sidecar stays ready):
mongodb-1: metrics ready=true  restarts=0
mongodb-1: mongodb ready=false restarts=9  lastReason=Completed

$ kubectl top pod -n nvsentinel | grep mongo
mongodb-0    341m    1621Mi
mongodb-1   1503m     589Mi     # pinned at the 1500m CPU limit
mongodb-2   1502m     481Mi     # pinned at the 1500m CPU limit

mongod connection count climbing past 150 on a flapping replica:

mongodb-1 NETWORK "Connection accepted" ... "connectionCount":153

Replicaset has no primary → remediation consumers crash

fault-quarantine in CrashLoopBackOff (7 restarts), fatal startup error:

level=ERROR msg="Application encountered a fatal error"
error="failed to create datastore: ... error connecting to database:
retrying ping to database HealthEventsDatabase timed out ...
server selection error: server selection timeout, current topology:
{ Type: ReplicaSetNoPrimary, Servers: [
  { Addr: mongodb-0...:27017, Type: RSSecondary },
  { Addr: mongodb-1...:27017, Type: Unknown, Last error: context deadline exceeded },
  { Addr: mongodb-2...:27017, Type: Unknown, Last error: context deadline exceeded } ] }"

node-drainer: 34 restarts.

Remediation outcome

cordoned (unschedulable) kwok nodes : 11
GPUReset CRs                        : 0
reset Jobs                          : 0

(For comparison, an earlier 5k run where MongoDB stayed healthy produced ~765
cordoned nodes and ~939 GPUReset CRs from the same class of fatal-event load.)

Root cause

1. Each of the ~1,600 pool connectors is a real platform-connector client that
   maintains an mTLS (X.509) connection to MongoDB. X.509 handshakes are
   CPU-expensive on the server side, and the steady connection count is high.
2. The mongodb container is capped at 1.5 CPU cores. The connection/handshake
   load pins mongod at that ceiling.
3. A CPU-starved mongod misses its readiness/liveness probe deadlines, so 2 of 3
   replicas are marked NotReady and restarted (lastReason=Completed = probe kill).
4. With a majority of members unavailable, the replicaSet cannot elect/keep a
   primary (ReplicaSetNoPrimary).
5. fault-quarantine / node-drainer require the datastore on startup and during
   steady operation; with no primary they crash-loop and never process the
   fatal events → no cordon, no GPUReset.

In short: the connector fan-in saturates the same CPU-limited MongoDB that the
remediation pipeline depends on, and starves it.

Impact

• Any deployment where the number of platform-connector clients (real or emulated)
  is large relative to MongoDB's CPU budget can lose the event-store primary,
  taking down remediation cluster-wide — not just at test time.
• For the harness specifically, event injection and the manual remediation-CR checks
  cannot pass while the platform connector pool is at high density, because injection
  and remediation compete for the same saturated store.

Suggested fixes / mitigations

1. Right-size MongoDB for the connector fan-in. Raise the mongodb container
   CPU limit (e.g. 1.5 → 4+ cores) and/or scale replicas/shard read load so the
   store can absorb ~1,600 mTLS clients. This is a genuine capacity-planning gap:
   the connector count per replica should be a documented, tested limit.
2. Reduce per-connection cost. Reuse TLS sessions / connection pooling on the
   connector side so 1,600 connectors don't each drive fresh X.509 handshakes;
   consider a connection-count cap per connector and server-side maxIncomingConnections.
3. Protect the primary. Ensure MongoDB probes tolerate transient CPU pressure
   (probe timeouts/thresholds) and/or give mongod a CPU request == limit (Guaranteed
   QoS) so it isn't throttled below what the probes need.
4. Decouple remediation from connector load. If practical, isolate the read/write
   path fault-quarantine/node-drainer use (dedicated read preference, separate
   connection budget) so a connector-driven spike can't starve remediation.

Component

Core Service

Steps to Reproduce

1. Run 5,000 KWOK nodes
2. Run around ~1,600 platform connector pod pool
3. Inject fatal/non fatal events in parallel

Expected symptom: mongodb-1/mongodb-2 drop to 1/2 Running and restart, CPU
pins at ~1500m, fault-quarantine goes CrashLoopBackOff, cordoned/GPUReset counts
stay near zero.

Environment

• NVSentinel version: v1.9.0
• Kubernetes version: v1.32.5
• Deployment method: Helm

Logs/Output

1. MongoDB replicas flapping — mongod container NotReady, restarting every 20–40s

$ kubectl get pods -n nvsentinel -l app.kubernetes.io/name=mongodb
NAME        READY   STATUS    RESTARTS       AGE
mongodb-0   2/2     Running   16 (11m ago)   14d
mongodb-1   1/2     Running   9  (40s ago)   14d
mongodb-2   1/2     Running   9  (19s ago)   14d

non-ready container is mongod itself (metrics sidecar stays ready), probe-killed:
mongodb-1: mongodb ready=false restarts=9 lastReason=Completed

2. mongod pinned at its 1500m CPU limit under the connector fan-in

$ kubectl top pod -n nvsentinel | grep mongo
mongodb-0    341m    1621Mi
mongodb-1   1503m     589Mi     # pinned at 1500m limit
mongodb-2   1502m     481Mi     # pinned at 1500m limit

connection count climbing on a flapping replica:
mongodb-1 NETWORK "Connection accepted" ... "connectionCount":153

3. ReplicaSet has NO primary → remediation consumers can't connect

fault-quarantine CrashLoopBackOff, fatal startup error:
level=ERROR msg="Application encountered a fatal error"
error="failed to create datastore: ... retrying ping to database HealthEventsDatabase timed out ...
server selection error: server selection timeout, current topology:
{ Type: ReplicaSetNoPrimary, Servers: [
  { Addr: mongodb-0...:27017, Type: RSSecondary },
  { Addr: mongodb-1...:27017, Type: Unknown, Last error: context deadline exceeded },
  { Addr: mongodb-2...:27017, Type: Unknown, Last error: context deadline exceeded } ] }"
node-drainer: 34 restarts

4. Resulting remediation outcome (events injected fine: 6,400/6,400 acked)

cordoned (unschedulable) kwok nodes : 11
GPUReset CRs                        : 0
reset Jobs                          : 0
(healthy-MongoDB 5k baseline for comparison: ~765 cordoned, ~939 GPUReset CRs)

[lalitadithya]

@nitz2407 , can you try by 2x'ing the MongoDB CPU and memory requests/limits?