fix(cloakserve): evict poll()-alive but CDP-dead Chrome instead of 502ing (ADR-0163 stage 0) - #1
Merged
Merged
Conversation
…2ing (ADR-0163 stage 0) During an olabrowser identity-lease rebuild, a client closes Chrome and immediately relaunches on the same fingerprint seed. Chrome tears down its DevTools/CDP socket seconds before the OS process exits, so process.poll() reports the corpse as alive and get_or_launch hands it back; handle_json_version then gets connection-refused on the CDP port and returned 502 WITHOUT evicting, so retry bursts kept hitting the same corpse. This is the entire session_infra failure bucket (~8% of scrapes; a 6h prod sample showed 3,241 dead-port 502s vs 3,243 warm 'already running' hits — ~100% warm-reuse failure across a rebuild). get_or_launch now probes the CDP port (a cheap 0.5s local TCP connect) before reusing a poll()-alive process, evicting and relaunching a corpse. The /json/version and /json/list handlers evict + retry once on a connection error as defense-in-depth. Eviction matches the pooled process by object identity, so a concurrent same-seed relaunch is never killed. 108 tests pass (6 new, mutation-verified). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix the
session_infracorpse race (ADR-0163 stage 0). During an olabrowser identity-lease rebuild, a client closes Chrome and immediately relaunches on the same fingerprint seed. Chrome tears down its DevTools/CDP socket seconds before the OS process exits, soprocess.poll()reports the corpse as alive andget_or_launchhands it back.handle_json_versionthen gets connection-refused on the CDP port and returned 502 without evicting the dead entry, so retry bursts kept hitting the same corpse.Impact: the entire
session_infrafailure bucket, ~8% of scrapes. A 6h prod sample showed 3,241 dead-port 502s vs 3,243 warm "already running" hits — warm reuse across a rebuild failed ~100% of the time.Fix
get_or_launchprobes the CDP port (a cheap 0.5s local TCP connect) before reusing apoll()-alive process; a corpse fails the probe and is evicted + relaunched./json/versionand/json/listevict + retry once on a connection error, so a process dying between the probe and the fetch becomes one cold relaunch instead of a persistent 502.Tests
108 pass (6 new). Mutation-verified: reverting only the source makes 5 of the new tests fail.
Out of scope
The olabrowser-side retry backoff is a separate follow-on; this fix stands alone — the corpse now becomes a clean cold launch, so retries succeed.