diff --git a/.jules/bolt.md b/.jules/bolt.md index 7fe37abc..a20501cf 100644 --- a/.jules/bolt.md +++ b/.jules/bolt.md @@ -56,3 +56,6 @@ ## 2025-10-30 - [Avoid HashMap Allocation in Tree Indexing] **Learning:** In the `cancel_subtree` function within `orch8-engine/src/evaluator.rs`, allocating a `HashMap>` to index the tree for a BFS traversal introduces massive hashing and memory allocation overhead on the cancellation hot path, resulting in O(N²) overall cost for deep trees when combining the allocation and iteration per node. **Action:** Replace dynamic parent-to-children index maps (`HashMap>`) with a flat `Vec<(ParentId, ChildId)>` sorted by parent ID. This allows using `.partition_point()` to perform O(log N) zero-allocation lookups for children on every visited node. +## 2026-06-24 - [DashMap Lock Contention & Key Allocation in Cascading Failures] +**Learning:** When a circuit breaker is already in an `Open` state during a failure storm, simply incrementing the failure count with a write lock via `.get_mut(q)` causes unnecessary `DashMap` contention across the shard. The `failure_count` isn't actively evaluated when the breaker is already `Open` (only the `opened_at` timestamp matters for the `HalfOpen` transition check). +**Action:** Always implement a fast-path read lock using `.get()` in high-throughput updates to a concurrent `DashMap` when mutations are functionally no-ops (like adding failures to an already open breaker). Return early to completely avoid the write lock. diff --git a/orch8-engine/src/circuit_breaker.rs b/orch8-engine/src/circuit_breaker.rs index bd092593..cfca371f 100644 --- a/orch8-engine/src/circuit_breaker.rs +++ b/orch8-engine/src/circuit_breaker.rs @@ -306,6 +306,16 @@ impl CircuitBreakerRegistry { let now = Utc::now(); let search = KeyRef(tenant_id, handler); let q: &dyn CircuitKey = &search; + + // ⚡ Bolt: Fast path read lock to avoid write-lock contention. If the circuit + // is already fully Open, there is no need to increment the failure count + // or acquire a write lock on the DashMap shard. + if let Some(breaker) = self.breakers.get(q) { + if breaker.state == BreakerState::Open { + return; + } + } + let mut tripped_snapshot = None; if let Some(mut breaker) = self.breakers.get_mut(q) {