docs(perf): S11 endurance soak — 4 h sustained real path, zero drift; API RSS leak found#182
Merged
Merged
Conversation
…I leak found 4 h paced run (720k events queued, ~47 eps delivered) through the full real path plus 12 rpm API reads: bounded lag (peak 2915 -> 0, no death spiral), bridge RSS/FD flat for 4 h, one live broker/Redis fault survived with an exactly-once 256-event batch replay via the journal guard, and zero cache drift (API metric == direct deduplicated CH sum to the kopeck). Driver-side accounting is honest: 94.8% of queued events were delivered (rdkafka flush timeout + queue expiry during the fault window); the serving path lost nothing that reached events.validated. Finding: API process RSS grows 175 MB -> 1.67 GB over the soak (bridge flat) — recorded in STATUS.md Known issues and filed as a GitHub issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DORA Metrics
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
S11 executed on the Mac stand: 4 h paced soak (720k events queued at 50 eps, ~47 eps delivered) through produce → Flink → bridge → ClickHouse, with steady API read traffic (12 rpm) and Redis push invalidation.
Best part: at t≈194 min a real (not injected) fault hit — Redis publish timeouts, one failed batch apply, Kafka group session timeout. Recovery was fully automatic: offsets held, the 256-event batch replayed, journal guard collapsed all 256 as duplicates. No loss, no double-apply,
apply_failures_total= 1.Finding: API process RSS grows without bound under steady load (175 MB → 1.67 GB over 4 h; bridge flat) — added to STATUS.md Known issues, GitHub issue filed separately.
Honest accounting: 94.8 % of queued events were delivered (driver-side rdkafka flush timeout + queue expiry during the fault window, fire-and-forget produce); everything that reached
events.validatedwas applied exactly once.Files
docs/perf/soak-s11-2026-07-10.md— full report (fault timeline, alert thresholds, follow-ups, reproduce)docs/STATUS.md— soak row in Proven, Known issues section, Next rolled forwarddocs/serving-bridge.md— endurance line in the throughput bulletDocs-only change.
🤖 Generated with Claude Code