Skip to content

stagehand-python@4.2.0a0.dev1528

@shrey150 shrey150 tagged this 22 Sep 19:45
## Summary

Restore Browse V3 network-capture behavior on Stagehand V4 through a
CLI-private CDP sidecar, without committing core Stagehand, its
protocol, or generated SDKs to a public network-event schema.

## Stack (#2872)

1. #2833 — exact Browse V3 baseline import
2. #2834 — Stagehand V4 runtime and standard command parity
3. #2869 — CLI-owned cursor overlay
4. **#2849 — CLI-private CDP sidecar; V3 network parity**
5. #2835 — remove `--return-xpath`; supported V3 parity/release
checkpoint
6. #2838 — eval and packaging integration
7. #2839 — managed Context names (fast-follow)
8. #2701 — shared Functions core consumer (fast-follow)

## Architecture

- Lazily open one CLI-private browser-level CDP WebSocket using the
endpoint already held by the initialized Stagehand client.
- Attach a flattened CDP session to the active V4 page target.
- Present the unchanged V3 network writer with the same `on` / `off` /
`send` session shape.
- On `network off`, remove listeners, send `Network.disable`, and detach
the page target.
- Keep the browser-level sidecar WebSocket alive across off/on cycles;
closing an auxiliary Browserbase socket during the live session can
disconnect the browser.
- Close the sidecar only during Browse session cleanup.
- Never log or return the signed browser WebSocket endpoint.

The request correlation and request/response JSON writer are inherited
from the V3 CLI. No public protocol schema, extension, SDK, or
generated-client changes are included. #2832 remains open outside the
landing stack for that separate API-design discussion.

## User-visible behavior

The V3 command and file surface is retained:

```bash
browse network on
browse network path
browse network clear
browse network off
```

Each request directory contains `request.json` and, when available,
`response.json`, including the existing treatment of POST bodies,
response bodies, failures, redirects, cache hits, and binary responses.

## E2E Test Matrix

Full post-propagation implementation verification ran against exact
clean #2849 head `adbe80d8bbfc337fea9dc38e41a88da18c287046`. Its
seven-file network patch has the same stable patch ID
(`f062ac789108ddf640bc912776dafa210411f50c`) as the previously
stress-tested head, so the deterministic V3/V4 and MSN/CNN evidence
below applies unchanged. Frozen install, fresh builds, focused/full
tests, and the real Browserbase lifecycle were rerun on `adbe80d8b`.
Current review-fix head `9887732b6f0c32cdef0966ebfd2ac2c454321d4b`
changes only the test helper timeout diagnostic; on that exact head, the
targeted network-capture tests passed 2/2 and Browse formatting, ESLint,
and TypeScript checks passed. The comparison CLI was the exact built V3
implementation at `7365a20d52955c10d72606f2e6ddd74791609d13`. Every CLI
flow used a unique daemon directory; no pre-existing daemon was reused
or stopped.

| Command / flow | Observed output | Confidence / sufficiency |
| --- | --- | --- |
| `pnpm install --frozen-lockfile`; build extension, local Stagehand
SDK, then `browse` | Frozen install and all three builds passed; the CLI
manifest was generated from the tested workspace. | Proves the clean
stacked head installs from its lockfile and the subprocess tests
exercised fresh V4 SDK/extension/CLI artifacts, not stale `dist` output.
|
| Built V3 vs built V4 deterministic fixture: `open → network on →
eval(GET, cached GET twice, POST, redirect, binary, abort) → path → off
→ path → clear → stop` | V3: 8 requests / 7 responses. V4: 8 requests /
7 responses. Evaluation results matched; normalized request/response
diff count was **0**; command-shape diff was `[]`; clear left 0 entries
for both. | Exact parity for the stable V3 command and on-disk contract.
Normalization was limited to request ID, timestamp/duration, HTTP
`Date`, and fixture origin/port. |
| Concurrent real CLI enables: two `browse network on` subprocesses
launched together, followed by one marked navigation | One subprocess
returned `{ enabled: true }`; the other returned `{ alreadyEnabled:
true, enabled: true }`. The marked navigation produced exactly 1 request
record with 1 unique request ID. | Proves overlapping `network on` calls
serialize and attach listeners once rather than duplicating capture. |
| Real CLI `on → navigate → off → on → navigate → off` without clearing
| The two records were numbered `000` and `001`; both URLs remained
present, and the SHA-256 of the first `request.json` was unchanged after
the second cycle. | Proves off/on resumes at the next on-disk counter
and does not overwrite retained captures. |
| Real CLI permissions and lifecycle | Capture directories were `0700`;
`request.json` and `response.json` were `0600`; `network clear` left 0
entries; `browse stop` reported stopped. | Proves the V3 private-file
contract and cleanup behavior through the built CLI. |
| Real Browserbase, two `network on → open Example Domain → off` cycles
in one session, then navigation with capture off | Both cycles captured
GET 200 and the complete Example Domain body; the Browserbase session
stayed stable; post-off `open --wait networkidle` worked; final status
was connected, initialized, and remote; stop completed. | Proves the
auxiliary sidecar survives real remote off/on while the owning Stagehand
browser remains usable. No session ID or signed endpoint is included
here. |
| Real Browserbase stress, V3 and V4, public MSN then CNN; each cycle
ran `on → path → open → scroll → collect 12s → off → path → navigate
while off → status → clear` | All 21 commands completed for each CLI. V3
captured MSN 415/401 and CNN 192/187 request/response records. V4
captured MSN 498/481 and CNN 194/187. Across all four cycles: 0
malformed request JSON, 0 malformed response JSON, 0 response-ID
mismatches, 0 new records after off, and 0 entries after clear. Both
sessions stayed remote/connected, stopped cleanly, and emitted no
WebSocket endpoint. | Proves bounded high-concurrency, cross-origin
Document/Script/Fetch/XHR/Image/Font/Media traffic. Live request counts
are intentionally not expected to match because sites and ad auctions
are nondeterministic; contract invariants and command shapes matched. |
| `pnpm --filter browse lint` | Passed formatting, ESLint, and
TypeScript checks at full-verification head `adbe80d8b`; passed again at
current review-fix head `9887732b6`. | Static support for both the fully
exercised implementation head and the exact current head. |
| `pnpm --filter browse test` with isolated daemon directory | At
full-verification head `adbe80d8b`: **27 files / 390 tests passed**;
focused network subset: **3 files / 30 tests passed**. At current
review-fix head `9887732b6`: targeted network-capture tests passed
**2/2**. | Covers sidecar routing, response bodies, listener teardown,
attach/detach, request-write races, counter reuse, and overlapping
enable serialization in addition to the real flows above; the
current-head rerun is scoped to the test-only diagnostic change. |

The deterministic artifact comparison includes method, URL/path,
request/response headers and bodies, status/status text, MIME type,
resource type, error shape, counter naming, and file modes. The live
stress test also observed GET/POST/OPTIONS, failures, base64 bodies,
cross-origin documents, and request-only records for traffic still in
flight at the bounded `off` point.

This matrix does not claim WebSocket-frame, SSE-message, service-worker,
or every out-of-process-iframe edge-case coverage; those are outside the
V3 JSON request/response file contract proven here.
Assets 2
Loading