feat(browse): add --navigate flag to download for browser-triggered file downloads by garrytan-agents · Pull Request #1355 · garrytan/gstack

garrytan-agents · 2026-05-07T05:58:24Z

Problem

The download command uses page.request.fetch() which creates a separate HTTP request outside the browser context. This fails for URLs that trigger actual browser downloads via:

Content-Disposition headers — server says "download this file"
CDN redirect chains — DDoS-Guard → CDN redirect that requires browser cookies
Cross-origin redirects — CORS blocks fetch() but browser navigation works

Concrete example: Anna's Archive fast downloads. The browser can login, get auth cookies, and navigate to a download URL — but page.request.fetch() can't follow the redirect chain through DDoS-Guard. The goto command says "Download is starting" but the file is never persisted.

Solution

New --navigate flag on the download command:

browse download "https://example.com/file.epub" /tmp/book.epub --navigate

When --navigate is set, instead of page.request.fetch(), the command:

Sets up page.waitForEvent("download") listener
Calls page.goto(url) with waitUntil: "commit" (since the page may redirect to trigger the download)
Captures the Playwright Download object
Calls download.saveAs(path) to persist the file
Infers content type from the suggested filename

This uses the browser's native download handling, which includes all cookies, proxy settings, and TLS fingerprint — exactly what sites like Anna's Archive, DDoS-Guard-protected CDNs, and authenticated download portals require.

Usage

# Old behavior (direct fetch, still default):
browse download "https://cdn.example.com/file.pdf" /tmp/file.pdf

# New: browser-triggered download:
browse download "https://protected-site.com/download/abc123" /tmp/file.epub --navigate

# Works with --base64 too:
browse download "https://protected-site.com/download/abc123" --navigate --base64

Testing

Tested end-to-end with Anna's Archive fast downloads through NordVPN SOCKS proxy:

Login via form → auth cookies set
download --navigate → file captured and saved
4.1MB EPUB validated

Changes

browse/src/write-commands.ts: Added --navigate strategy (Strategy 2) to download command
browse/src/commands.ts: Updated help text to document --navigate flag

^{Need help on this PR? Tag @codesmith with what you need.}

Let Codesmith autofix CI failures and bot reviews

…ggered downloads The download command currently uses page.request.fetch() which creates a separate HTTP request. This fails for URLs that trigger actual browser downloads via: - Content-Disposition headers - CDN redirect chains (e.g. DDoS-Guard protected sites) - Cross-origin redirects that break CORS The new --navigate flag uses page.goto() + page.waitForEvent('download') + download.saveAs() to capture files that only work through native browser download handling. Usage: download <url> [path] --navigate Example: download 'https://annas-archive.gl/fast_download/abc123/0/0' /tmp/book.epub --navigate This enables downloads from DDoS-Guard protected CDNs (Anna's Archive, libgen mirrors) where the browser has authenticated cookies but page.request.fetch() can't follow the redirect chain.

@garrytan-agents

…+ webdriver-only stealth (#1363) * feat(browse): SOCKS5 bridge with auth + cred redaction helper Adds browse/src/socks-bridge.ts: a 127.0.0.1-only SOCKS5 listener that accepts unauthenticated connections from Chromium and relays them through an authenticated upstream proxy. Chromium does not prompt for SOCKS5 auth at launch, so this bridge is the workaround for using auth-required residential SOCKS5 upstreams. - startSocksBridge({ upstream, port: 0 }) → ephemeral 127.0.0.1 listener - testUpstream({ upstream, retries: 3, backoffMs: 500, budgetMs: 5000 }) pre-flight that connects to a known endpoint (default 1.1.1.1:443) - Stream-error policy: kill affected client + upstream sockets on any error mid-stream; no transport retries (a transport-layer retry can corrupt browser traffic) Adds browse/src/proxy-redact.ts: single source of truth for redacting credentials in any logged proxy URL or upstream config. Every code path that prints proxy config goes through this helper. Adds the socks npm dep (~30KB) and 16 tests covering: 127.0.0.1-only bind, byte-for-byte round trip through the bridge, auth rejection, mid-stream upstream drop kills client conn, listener teardown, testUpstream success + retry-exhaust paths, redaction of every credential shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): --proxy and --headed flags wire bridge into daemon Adds the global --proxy <url> and --headed flags to the browse CLI. Resolves cred policy and routes the daemon launch through the SOCKS5 bridge (or pass-through for HTTP/HTTPS) before chromium.launch(). CLI (cli.ts): - extractGlobalFlags() strips --proxy/--headed from argv, parses URL via Node URL class, validates D9 cred-mixing (env BROWSE_PROXY_USER/PASS + URL creds → exit 1 with hint), composes canonical proxy URL with resolved creds, computes a stable configHash for daemon-mismatch - ensureServer() now reads existing daemon's configHash from state file and refuses (exit 1 with disconnect hint) if --proxy/--headed mismatch the existing daemon. No silent restart that would drop tab state. - All proxy-related stderr lines go through redactProxyUrl proxy-config.ts (new): - parseProxyConfig() — URL parser + D9 cred-mixing detector + scheme allowlist - computeConfigHash() — stable hash of (proxy URL minus creds + headed flag) - toUpstreamConfig() — map ParsedProxyConfig → socks-bridge.UpstreamConfig Server (server.ts): - Reads BROWSE_PROXY_URL at startup; for SOCKS5+auth, runs testUpstream pre-flight (5s budget, 3 retries, 500ms backoff) and exits 1 on failure with redacted error - Spawns startSocksBridge() on 127.0.0.1:<ephemeral> and points Chromium at it via socks5://127.0.0.1:<port> - HTTP/HTTPS or unauth SOCKS5 → pass-through to chromium.launch proxy.server (with username/password if present) - State file gains optional configHash for daemon-mismatch check - Bridge tears down via process.on('exit') Browser manager (browser-manager.ts): - New setProxyConfig({ server, username, password }) called by server.ts before launch - chromium.launch() and both launchPersistentContext sites pass the proxy config through when set Tests: 22 new across proxy-config (parse + cred-mixing + hash stability) and extractGlobalFlags (flag stripping + cred-mixing rejection + cred rotation hash stability + redaction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): Xvfb auto-spawn with PID + start-time validation Adds browse/src/xvfb.ts: a Linux-only Xvfb auto-spawn module for running headed Chromium in containers without DISPLAY. The module walks a display range to pick a free one (never hardcodes :99) and validates orphan PIDs by BOTH /proc/<pid>/cmdline matching 'Xvfb' AND start-time matching the recorded value before sending any signal. Defends against PID reuse — refuses to kill anything that doesn't match both checks. - shouldSpawnXvfb(env, platform) — pure decision: skip on macOS/Windows, on Linux skip when DISPLAY or WAYLAND_DISPLAY is set (codex F2) - pickFreeDisplay(99..120) — probes via xdpyinfo - spawnXvfb(display) — returns { pid, startTime, display } handle - isOurXvfb(pid, startTime) — both-checks validator - cleanupXvfb(state) — best-effort, validates ownership before SIGTERM Wired into server.ts startup: when shouldSpawnXvfb says yes, picks a free display, spawns Xvfb, sets DISPLAY for chromium.launchHeaded, and records xvfbPid/xvfbStartTime/xvfbDisplay in the state file. Cleanup runs on process.on('exit'). The CLI's disconnect path also runs cleanupXvfb() in the force-cleanup branch when the server is dead. Disconnect now applies to any non-default daemon (headed mode OR configHash-tagged daemon — i.e. one started with --proxy/--headed), not just headed mode. Adds xvfb + x11-utils to .github/docker/Dockerfile.ci so CI exercises the Linux container --headed path on every run. Without it the most common production path would go untested. Tests: 17 new across decision logic, PID validation defenses (cmdline mismatch, start-time mismatch), no-op safety on bad inputs, and a Linux+Xvfb-installed gate for the spawn → validate → cleanup round trip. Tests skip on macOS/Windows automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): webdriver-mask stealth + Chromium-through-bridge e2e D7 (codex narrowing): mask navigator.webdriver only via addInitScript. The wintermute approach (fake plugins=[1..5], fake languages=['en-US', 'en'], stub window.chrome) is intentionally NOT applied — modern fingerprinters check consistency between plugins.length, languages, userAgent, and platform, and synthesizing fixed values can flag MORE bot-like, not less. The honest minimum is webdriver, which Chromium exposes as a known automation tell. Adds browse/src/stealth.ts: single source of truth for the stealth init script and launch args. Both browser-manager.launch() (headless) and launchHeaded() (persistent context with extension) call applyStealth(context) and pass STEALTH_LAUNCH_ARGS into chromium.launch. The pre-existing launchHeaded stealth that did fake plugins/languages is removed for the same reason. The cdc_/__webdriver runtime cleanup and Permissions API patch are kept — they remove automation-injected artifacts, not synthesize fake natural-browser values. Adds bridge-chromium-e2e.test.ts (codex F3): the test that proves the FEATURE works. Real Chromium with proxy.server = 'socks5://127.0.0.1: <bridgePort>' navigates to a local HTTP fixture; the auth upstream's connect counter and the HTTP fixture's hit counter both increment, proving traffic actually traversed bridge → auth-upstream → destination. Without this test, we could ship a working byte-relay and a broken Chromium integration and never know. Adds bridge-port-restart.test.ts (codex F1, reframed): old test assumed two daemons coexist, which contradicts D2 single-daemon model. Reframed as restart-then-restart, asserting fresh ephemeral ports (never the hardcoded 1090) on each spin-up. Adds stealth-webdriver.test.ts: navigator.webdriver=false in both fresh contexts and persistent contexts; navigator.plugins/languages are NOT replaced with the wintermute fake list (D7 verification). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(gstack): generate llms.txt — single-file capability index for AI agents Adds scripts/gen-llms-txt.ts: produces gstack/llms.txt at repo root, indexing every skill (47), every browse command (75), and design commands when the design CLI is present. Per the llmstxt.org convention, agents can read one file to learn what gstack offers instead of crawling 47 SKILL.md files. Sources: - skill SKILL.md.tmpl frontmatter (name + description block scalar) - browse/src/commands.ts COMMAND_DESCRIPTIONS (sorted by category) - design/src/commands.ts COMMAND_DESCRIPTIONS if present (best-effort) Wired into scripts/gen-skill-docs.ts as a post-step so it regenerates on every `bun run gen:skill-docs` (the same script that re-emits all SKILL.md files). Failures are non-fatal warnings, not build breaks — the generator never blocks SKILL.md regen. Strict mode (--strict, also used by tests) throws when a skill is missing name or description in its frontmatter, catching missing metadata before it ships. Tests: shape (top-level sections, sort order, single-line summary discipline), every-skill-and-command-appears, strict-mode rejection of incomplete frontmatter, and freshness check that the committed gstack/llms.txt matches what the generator produces now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): --navigate flag on download for browser-triggered files Adds the --navigate strategy from community PR #1355 (originally from @garrytan-agents). When set, download navigates to the URL with waitUntil:'commit' and captures the resulting browser download via page.waitForEvent('download'), then saves via download.saveAs(). Handles URLs that trigger files via Content-Disposition headers, multi-hop CDN redirects requiring browser cookies, or anti-bot CDN chains where page.request.fetch() can't follow the auth/redirect chain. Defaults still use the existing direct-fetch strategy. --navigate is opt-in. Goes through the same validateNavigationUrl SSRF gate as goto, so download --navigate cannot reach IPv4 metadata endpoints (AWS IMDSv1, GCP/Azure equivalents) or arbitrary internal hosts. Inferred content type from suggested filename for common extensions (epub, pdf, zip, gz, mp3/mp4, jpg/jpeg/png, txt, html, json) — falls back to application/octet-stream. Same 200MB cap as Strategy 1. Frames the use case generically (anti-bot CDN, Content-Disposition, redirect chains) rather than naming any specific site, per project voice rules. Co-Authored-By: @garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v1.28.0.0 — browse SKILL section + VERSION + CHANGELOG VERSION 1.27.1.0 → 1.28.0.0 (MINOR — substantial new capability: five new flags/features, ~600 LOC added, new socks dep, multiple new modules). browse/SKILL.md.tmpl: new "Headed Mode + Proxy + Anti-Bot Sites" section between User Handoff and Snapshot Flags. Documents --headed (auto-Xvfb on Linux), --proxy (with embedded SOCKS5 bridge for auth), download --navigate, the cred-mixing policy, daemon-discipline (refuse-on-mismatch), the narrowed webdriver-only stealth, container support caveats, and the fail-fast/no-retry failure modes. CHANGELOG entry follows the release-summary format from CLAUDE.md: two-line headline, lead paragraph, "The numbers that matter" table tied to specific test files that prove each capability, "What this means for AI agents" closing tied to a real workflow shift, then itemized Added/Changed/Fixed/For-contributors sections. Browse SKILL.md regenerated via bun run gen:skill-docs. gstack/llms.txt regenerated automatically from the same pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(browse): integration coverage for daemon mismatch + proxy fail-fast Adds two integration tests that exercise the full process boundary, not just the module-level wiring. daemon-mismatch-refuse.test.ts (D2): - Stubs a healthy state file with a fake configHash and a fake /health HTTP server, runs the actual cli.ts binary with a mismatching --proxy, asserts exit 1 + 'different config' / 'browse disconnect' hint in stderr. - Same shape with the plain-daemon-meets---headed case. - Positive case: matching configHash → CLI does NOT emit the mismatch hint (regardless of whether the actual command succeeds). server-proxy-fail-fast.test.ts: - Starts the rejecting SOCKS5 upstream, spawns server.ts with BROWSE_PROXY_URL pointing at it, BROWSE_HEADLESS_SKIP=1 to skip Chromium launch. - Asserts exit 1, 'FAIL upstream' in stderr (testUpstream pre-flight ran), no raw credential leakage in any output (redaction works on the failure path), and exit within 30s upper bound. Both tests use the existing spawn-bun-cli pattern from commands.test.ts so they run on the same CI infrastructure as the rest of the bun test suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(gen-skill-docs): keep module sync so test require() still works Two regressions caught by the full test suite after the v1.28.0.0 landing pass: 1) package.json version mismatch — VERSION was bumped to 1.28.0.0 but package.json still pinned to 1.27.1.0. test/gen-skill-docs.test.ts asserts they match. 2) Top-level await in scripts/gen-llms-txt.ts (CLI entry block) and scripts/gen-skill-docs.ts (post-step) made gen-skill-docs an async module. test/gen-skill-docs.test.ts uses require() to pull extractVoiceTriggers/processVoiceTriggers from gen-skill-docs, which Bun rejects on async modules with: "TypeError: require() async module ... unsupported. use 'await import()' instead." Fix: wrap the await blocks in void IIFEs so the modules remain sync from a require() perspective. After fix: all 379 gen-skill-docs tests pass, all 77 new feature tests pass (3 skipped on macOS — Linux+Xvfb gates). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(browse): apply codex adversarial findings on the new lifecycle Codex outside-voice review caught five real production-failure modes in the v1.28.0.0 proxy/headed lifecycle. Fixed: 1) `browse disconnect` skip-graceful for proxy-only daemons (browse/src/cli.ts). The graceful /command POST went out with stray `domains,` shorthand and (even fixed) the server's disconnect handler only tears down headed mode — proxy-only daemons returned 200 "Not in headed mode" while leaving the bridge running. Now disconnect short-circuits to force-cleanup for non-headed daemons, which kicks process.on('exit') in server.ts to close the bridge + Xvfb. 2) sendCommand crash retry preserves --proxy / --headed (browse/src/cli.ts). The ECONNRESET retry path called startServer() with no extraEnv, silently dropping the proxied flags. A daemon that died mid-command would silently restart in default direct/headless mode and bypass the SOCKS bridge. Now reapplies BROWSE_PROXY_URL, BROWSE_HEADED, and BROWSE_CONFIG_HASH from the resolved global flags. 3) `connect` honors --proxy (browse/src/cli.ts). The headed-mode `connect` command built its own serverEnv that didn't include BROWSE_PROXY_URL, so `browse --proxy <url> connect` launched headed Chromium without the proxy. Now threads proxyUrl + configHash into the connect serverEnv. 4) SOCKS5 bridge handles fragmented TCP frames (browse/src/socks-bridge.ts). Previously used once('data') and parsed each chunk as a complete SOCKS5 frame — TCP doesn't preserve message boundaries and split greetings/CONNECT requests caused intermittent handshake failures. Replaced with a single state machine that buffers chunks and uses size predicates on the SOCKS5 header to know when a complete frame has arrived. Pauses the client socket during upstream connect and replays any remainder bytes into the upstream on success. 5) Xvfb cleanup-then-state-delete ordering (browse/src/server.ts). emergencyCleanup() previously deleted the state file BEFORE any Xvfb cleanup could read it, orphaning Xvfb on uncaughtException / unhandledRejection. Now reads the state file first, calls cleanupXvfb() (which validates cmdline + start-time before kill), then deletes the state file. Adds a regression test for #4: writes the SOCKS5 greeting + CONNECT one byte at a time with 5ms ticks, asserts a clean round trip after the fragmented handshake. Codex's sixth finding (bridge advertises NO_AUTH on 127.0.0.1, so any co-located process can use the authenticated upstream) is documented as a known limitation — gstack's threat model assumes single-user hosts. Adding bridge-side auth is a separate change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update BROWSER.md + TODOS.md for v1.28.0.0 BROWSER.md picks up a "Headed mode + proxy + browser-native downloads (v1.28.0.0)" subsection inside Real-browser mode plus the new source-map entries (socks-bridge.ts, proxy-config.ts, proxy-redact.ts, xvfb.ts, stealth.ts). TODOS.md anti-bot-stealth item updated to reflect the v1.28 narrowing — the "fake plugins" line is no longer accurate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): include bun.lock in image build for deterministic install CI evals all failed on PR #1363 with: error: Could not resolve: "smart-buffer". Maybe you need to "bun install"? error: Could not resolve: "ip-address". Maybe you need to "bun install"? at /opt/node_modules_cache/socks/build/client/socksclient.js:15 The cached node_modules layer in the pre-baked Docker image had `socks` (the new dep) but was missing its transitive deps (smart-buffer, ip-address). The image build copied only package.json into the build context — without bun.lock, `bun install` resolved a different tree than local `bun install` did, dropping required transitive deps. Reproduces locally as 229 packages (correct) when bun.lock is present or absent. Why CI diverged isn't fully understood — possibly Docker layer cache reuse across image rebuilds — but the deterministic fix is to include the lockfile in the image build context and use `--frozen-lockfile`, matching what every CI doc recommends. Changes: - .github/docker/Dockerfile.ci: COPY bun.lock alongside package.json, switch `bun install` → `bun install --frozen-lockfile` so any future lockfile drift fails loudly during image build instead of producing a partially-installed cache that breaks downstream eval jobs. - .github/workflows/evals.yml: include bun.lock in the image-tag hash so adding/removing a dep invalidates the image, AND copy bun.lock into the docker context alongside package.json. - .github/workflows/evals-periodic.yml: same updates. - .github/workflows/ci-image.yml: rebuild trigger now fires on bun.lock changes too; build context includes bun.lock. Image hash changes → fresh image gets built on next CI run → install matches the lockfile exactly → no missing transitive deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): use hardlink copy instead of symlink for node_modules cache After the bun.lock fix landed, the eval matrix STILL failed identically: Could not resolve: "smart-buffer" / "ip-address" at /opt/node_modules_cache/socks/build/client/socksclient.js But the hash-tagged image actually contains smart-buffer + ip-address + socks all flat in /opt/node_modules_cache (verified by pulling and inspecting the image). 207 packages, all present. Root cause: the workflow used `ln -s /opt/node_modules_cache node_modules` to restore deps. Bun build (and Node module resolution generally) walks a file's realpath to find sibling deps. From the symlinked /workspace/node_modules/socks/build/client/socksclient.js, realpath resolves to /opt/node_modules_cache/socks/build/client/socksclient.js, and walking up to find a node_modules/smart-buffer dir fails — there's no `node_modules` segment in the realpath. Switch `ln -s` → `cp -al` (hardlink-copy). Each file in the cache becomes a hardlink at /workspace/node_modules/<pkg>, sharing inodes (no data copy). Realpath of /workspace/node_modules/socks/.../socksclient.js stays inside /workspace/node_modules, so sibling deps resolve correctly. Speed is comparable to symlink — `cp -al` on ~200 packages on tmpfs is sub-second. Same caching story preserved. Both evals.yml and evals-periodic.yml updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): cp -r instead of cp -al — /opt and /workspace are different filesystems The hardlink-copy fix landed and immediately broke with: cp: cannot create hard link 'node_modules/<file>' to '/opt/node_modules_cache/<file>': Invalid cross-device link GitHub Actions runners mount the workspace volume at /workspace (overlay-fs layered onto the runner image), and /opt is the runner image's own filesystem. Cross-filesystem hardlinks aren't supported. Switch `cp -al` → `cp -r`. Cost: ~5s for ~200 packages of small JS files vs ~0s for the broken symlink. Still cheaper than the ~15s `bun install` fallback. Realpath of /workspace/node_modules/<pkg>/... stays inside /workspace, so bun build's sibling-dep resolution works. Both evals.yml and evals-periodic.yml updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

garrytan · 2026-05-09T07:36:06Z

Closing as superseded — browse --navigate already shipped in v1.28.0.0 (#1363). Thanks for the contribution!

@garrytan-agents

…+ webdriver-only stealth (garrytan#1363) * feat(browse): SOCKS5 bridge with auth + cred redaction helper Adds browse/src/socks-bridge.ts: a 127.0.0.1-only SOCKS5 listener that accepts unauthenticated connections from Chromium and relays them through an authenticated upstream proxy. Chromium does not prompt for SOCKS5 auth at launch, so this bridge is the workaround for using auth-required residential SOCKS5 upstreams. - startSocksBridge({ upstream, port: 0 }) → ephemeral 127.0.0.1 listener - testUpstream({ upstream, retries: 3, backoffMs: 500, budgetMs: 5000 }) pre-flight that connects to a known endpoint (default 1.1.1.1:443) - Stream-error policy: kill affected client + upstream sockets on any error mid-stream; no transport retries (a transport-layer retry can corrupt browser traffic) Adds browse/src/proxy-redact.ts: single source of truth for redacting credentials in any logged proxy URL or upstream config. Every code path that prints proxy config goes through this helper. Adds the socks npm dep (~30KB) and 16 tests covering: 127.0.0.1-only bind, byte-for-byte round trip through the bridge, auth rejection, mid-stream upstream drop kills client conn, listener teardown, testUpstream success + retry-exhaust paths, redaction of every credential shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): --proxy and --headed flags wire bridge into daemon Adds the global --proxy <url> and --headed flags to the browse CLI. Resolves cred policy and routes the daemon launch through the SOCKS5 bridge (or pass-through for HTTP/HTTPS) before chromium.launch(). CLI (cli.ts): - extractGlobalFlags() strips --proxy/--headed from argv, parses URL via Node URL class, validates D9 cred-mixing (env BROWSE_PROXY_USER/PASS + URL creds → exit 1 with hint), composes canonical proxy URL with resolved creds, computes a stable configHash for daemon-mismatch - ensureServer() now reads existing daemon's configHash from state file and refuses (exit 1 with disconnect hint) if --proxy/--headed mismatch the existing daemon. No silent restart that would drop tab state. - All proxy-related stderr lines go through redactProxyUrl proxy-config.ts (new): - parseProxyConfig() — URL parser + D9 cred-mixing detector + scheme allowlist - computeConfigHash() — stable hash of (proxy URL minus creds + headed flag) - toUpstreamConfig() — map ParsedProxyConfig → socks-bridge.UpstreamConfig Server (server.ts): - Reads BROWSE_PROXY_URL at startup; for SOCKS5+auth, runs testUpstream pre-flight (5s budget, 3 retries, 500ms backoff) and exits 1 on failure with redacted error - Spawns startSocksBridge() on 127.0.0.1:<ephemeral> and points Chromium at it via socks5://127.0.0.1:<port> - HTTP/HTTPS or unauth SOCKS5 → pass-through to chromium.launch proxy.server (with username/password if present) - State file gains optional configHash for daemon-mismatch check - Bridge tears down via process.on('exit') Browser manager (browser-manager.ts): - New setProxyConfig({ server, username, password }) called by server.ts before launch - chromium.launch() and both launchPersistentContext sites pass the proxy config through when set Tests: 22 new across proxy-config (parse + cred-mixing + hash stability) and extractGlobalFlags (flag stripping + cred-mixing rejection + cred rotation hash stability + redaction). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): Xvfb auto-spawn with PID + start-time validation Adds browse/src/xvfb.ts: a Linux-only Xvfb auto-spawn module for running headed Chromium in containers without DISPLAY. The module walks a display range to pick a free one (never hardcodes :99) and validates orphan PIDs by BOTH /proc/<pid>/cmdline matching 'Xvfb' AND start-time matching the recorded value before sending any signal. Defends against PID reuse — refuses to kill anything that doesn't match both checks. - shouldSpawnXvfb(env, platform) — pure decision: skip on macOS/Windows, on Linux skip when DISPLAY or WAYLAND_DISPLAY is set (codex F2) - pickFreeDisplay(99..120) — probes via xdpyinfo - spawnXvfb(display) — returns { pid, startTime, display } handle - isOurXvfb(pid, startTime) — both-checks validator - cleanupXvfb(state) — best-effort, validates ownership before SIGTERM Wired into server.ts startup: when shouldSpawnXvfb says yes, picks a free display, spawns Xvfb, sets DISPLAY for chromium.launchHeaded, and records xvfbPid/xvfbStartTime/xvfbDisplay in the state file. Cleanup runs on process.on('exit'). The CLI's disconnect path also runs cleanupXvfb() in the force-cleanup branch when the server is dead. Disconnect now applies to any non-default daemon (headed mode OR configHash-tagged daemon — i.e. one started with --proxy/--headed), not just headed mode. Adds xvfb + x11-utils to .github/docker/Dockerfile.ci so CI exercises the Linux container --headed path on every run. Without it the most common production path would go untested. Tests: 17 new across decision logic, PID validation defenses (cmdline mismatch, start-time mismatch), no-op safety on bad inputs, and a Linux+Xvfb-installed gate for the spawn → validate → cleanup round trip. Tests skip on macOS/Windows automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): webdriver-mask stealth + Chromium-through-bridge e2e D7 (codex narrowing): mask navigator.webdriver only via addInitScript. The wintermute approach (fake plugins=[1..5], fake languages=['en-US', 'en'], stub window.chrome) is intentionally NOT applied — modern fingerprinters check consistency between plugins.length, languages, userAgent, and platform, and synthesizing fixed values can flag MORE bot-like, not less. The honest minimum is webdriver, which Chromium exposes as a known automation tell. Adds browse/src/stealth.ts: single source of truth for the stealth init script and launch args. Both browser-manager.launch() (headless) and launchHeaded() (persistent context with extension) call applyStealth(context) and pass STEALTH_LAUNCH_ARGS into chromium.launch. The pre-existing launchHeaded stealth that did fake plugins/languages is removed for the same reason. The cdc_/__webdriver runtime cleanup and Permissions API patch are kept — they remove automation-injected artifacts, not synthesize fake natural-browser values. Adds bridge-chromium-e2e.test.ts (codex F3): the test that proves the FEATURE works. Real Chromium with proxy.server = 'socks5://127.0.0.1: <bridgePort>' navigates to a local HTTP fixture; the auth upstream's connect counter and the HTTP fixture's hit counter both increment, proving traffic actually traversed bridge → auth-upstream → destination. Without this test, we could ship a working byte-relay and a broken Chromium integration and never know. Adds bridge-port-restart.test.ts (codex F1, reframed): old test assumed two daemons coexist, which contradicts D2 single-daemon model. Reframed as restart-then-restart, asserting fresh ephemeral ports (never the hardcoded 1090) on each spin-up. Adds stealth-webdriver.test.ts: navigator.webdriver=false in both fresh contexts and persistent contexts; navigator.plugins/languages are NOT replaced with the wintermute fake list (D7 verification). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(gstack): generate llms.txt — single-file capability index for AI agents Adds scripts/gen-llms-txt.ts: produces gstack/llms.txt at repo root, indexing every skill (47), every browse command (75), and design commands when the design CLI is present. Per the llmstxt.org convention, agents can read one file to learn what gstack offers instead of crawling 47 SKILL.md files. Sources: - skill SKILL.md.tmpl frontmatter (name + description block scalar) - browse/src/commands.ts COMMAND_DESCRIPTIONS (sorted by category) - design/src/commands.ts COMMAND_DESCRIPTIONS if present (best-effort) Wired into scripts/gen-skill-docs.ts as a post-step so it regenerates on every `bun run gen:skill-docs` (the same script that re-emits all SKILL.md files). Failures are non-fatal warnings, not build breaks — the generator never blocks SKILL.md regen. Strict mode (--strict, also used by tests) throws when a skill is missing name or description in its frontmatter, catching missing metadata before it ships. Tests: shape (top-level sections, sort order, single-line summary discipline), every-skill-and-command-appears, strict-mode rejection of incomplete frontmatter, and freshness check that the committed gstack/llms.txt matches what the generator produces now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(browse): --navigate flag on download for browser-triggered files Adds the --navigate strategy from community PR garrytan#1355 (originally from @garrytan-agents). When set, download navigates to the URL with waitUntil:'commit' and captures the resulting browser download via page.waitForEvent('download'), then saves via download.saveAs(). Handles URLs that trigger files via Content-Disposition headers, multi-hop CDN redirects requiring browser cookies, or anti-bot CDN chains where page.request.fetch() can't follow the auth/redirect chain. Defaults still use the existing direct-fetch strategy. --navigate is opt-in. Goes through the same validateNavigationUrl SSRF gate as goto, so download --navigate cannot reach IPv4 metadata endpoints (AWS IMDSv1, GCP/Azure equivalents) or arbitrary internal hosts. Inferred content type from suggested filename for common extensions (epub, pdf, zip, gz, mp3/mp4, jpg/jpeg/png, txt, html, json) — falls back to application/octet-stream. Same 200MB cap as Strategy 1. Frames the use case generically (anti-bot CDN, Content-Disposition, redirect chains) rather than naming any specific site, per project voice rules. Co-Authored-By: @garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v1.28.0.0 — browse SKILL section + VERSION + CHANGELOG VERSION 1.27.1.0 → 1.28.0.0 (MINOR — substantial new capability: five new flags/features, ~600 LOC added, new socks dep, multiple new modules). browse/SKILL.md.tmpl: new "Headed Mode + Proxy + Anti-Bot Sites" section between User Handoff and Snapshot Flags. Documents --headed (auto-Xvfb on Linux), --proxy (with embedded SOCKS5 bridge for auth), download --navigate, the cred-mixing policy, daemon-discipline (refuse-on-mismatch), the narrowed webdriver-only stealth, container support caveats, and the fail-fast/no-retry failure modes. CHANGELOG entry follows the release-summary format from CLAUDE.md: two-line headline, lead paragraph, "The numbers that matter" table tied to specific test files that prove each capability, "What this means for AI agents" closing tied to a real workflow shift, then itemized Added/Changed/Fixed/For-contributors sections. Browse SKILL.md regenerated via bun run gen:skill-docs. gstack/llms.txt regenerated automatically from the same pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(browse): integration coverage for daemon mismatch + proxy fail-fast Adds two integration tests that exercise the full process boundary, not just the module-level wiring. daemon-mismatch-refuse.test.ts (D2): - Stubs a healthy state file with a fake configHash and a fake /health HTTP server, runs the actual cli.ts binary with a mismatching --proxy, asserts exit 1 + 'different config' / 'browse disconnect' hint in stderr. - Same shape with the plain-daemon-meets---headed case. - Positive case: matching configHash → CLI does NOT emit the mismatch hint (regardless of whether the actual command succeeds). server-proxy-fail-fast.test.ts: - Starts the rejecting SOCKS5 upstream, spawns server.ts with BROWSE_PROXY_URL pointing at it, BROWSE_HEADLESS_SKIP=1 to skip Chromium launch. - Asserts exit 1, 'FAIL upstream' in stderr (testUpstream pre-flight ran), no raw credential leakage in any output (redaction works on the failure path), and exit within 30s upper bound. Both tests use the existing spawn-bun-cli pattern from commands.test.ts so they run on the same CI infrastructure as the rest of the bun test suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(gen-skill-docs): keep module sync so test require() still works Two regressions caught by the full test suite after the v1.28.0.0 landing pass: 1) package.json version mismatch — VERSION was bumped to 1.28.0.0 but package.json still pinned to 1.27.1.0. test/gen-skill-docs.test.ts asserts they match. 2) Top-level await in scripts/gen-llms-txt.ts (CLI entry block) and scripts/gen-skill-docs.ts (post-step) made gen-skill-docs an async module. test/gen-skill-docs.test.ts uses require() to pull extractVoiceTriggers/processVoiceTriggers from gen-skill-docs, which Bun rejects on async modules with: "TypeError: require() async module ... unsupported. use 'await import()' instead." Fix: wrap the await blocks in void IIFEs so the modules remain sync from a require() perspective. After fix: all 379 gen-skill-docs tests pass, all 77 new feature tests pass (3 skipped on macOS — Linux+Xvfb gates). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(browse): apply codex adversarial findings on the new lifecycle Codex outside-voice review caught five real production-failure modes in the v1.28.0.0 proxy/headed lifecycle. Fixed: 1) `browse disconnect` skip-graceful for proxy-only daemons (browse/src/cli.ts). The graceful /command POST went out with stray `domains,` shorthand and (even fixed) the server's disconnect handler only tears down headed mode — proxy-only daemons returned 200 "Not in headed mode" while leaving the bridge running. Now disconnect short-circuits to force-cleanup for non-headed daemons, which kicks process.on('exit') in server.ts to close the bridge + Xvfb. 2) sendCommand crash retry preserves --proxy / --headed (browse/src/cli.ts). The ECONNRESET retry path called startServer() with no extraEnv, silently dropping the proxied flags. A daemon that died mid-command would silently restart in default direct/headless mode and bypass the SOCKS bridge. Now reapplies BROWSE_PROXY_URL, BROWSE_HEADED, and BROWSE_CONFIG_HASH from the resolved global flags. 3) `connect` honors --proxy (browse/src/cli.ts). The headed-mode `connect` command built its own serverEnv that didn't include BROWSE_PROXY_URL, so `browse --proxy <url> connect` launched headed Chromium without the proxy. Now threads proxyUrl + configHash into the connect serverEnv. 4) SOCKS5 bridge handles fragmented TCP frames (browse/src/socks-bridge.ts). Previously used once('data') and parsed each chunk as a complete SOCKS5 frame — TCP doesn't preserve message boundaries and split greetings/CONNECT requests caused intermittent handshake failures. Replaced with a single state machine that buffers chunks and uses size predicates on the SOCKS5 header to know when a complete frame has arrived. Pauses the client socket during upstream connect and replays any remainder bytes into the upstream on success. 5) Xvfb cleanup-then-state-delete ordering (browse/src/server.ts). emergencyCleanup() previously deleted the state file BEFORE any Xvfb cleanup could read it, orphaning Xvfb on uncaughtException / unhandledRejection. Now reads the state file first, calls cleanupXvfb() (which validates cmdline + start-time before kill), then deletes the state file. Adds a regression test for garrytan#4: writes the SOCKS5 greeting + CONNECT one byte at a time with 5ms ticks, asserts a clean round trip after the fragmented handshake. Codex's sixth finding (bridge advertises NO_AUTH on 127.0.0.1, so any co-located process can use the authenticated upstream) is documented as a known limitation — gstack's threat model assumes single-user hosts. Adding bridge-side auth is a separate change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update BROWSER.md + TODOS.md for v1.28.0.0 BROWSER.md picks up a "Headed mode + proxy + browser-native downloads (v1.28.0.0)" subsection inside Real-browser mode plus the new source-map entries (socks-bridge.ts, proxy-config.ts, proxy-redact.ts, xvfb.ts, stealth.ts). TODOS.md anti-bot-stealth item updated to reflect the v1.28 narrowing — the "fake plugins" line is no longer accurate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): include bun.lock in image build for deterministic install CI evals all failed on PR garrytan#1363 with: error: Could not resolve: "smart-buffer". Maybe you need to "bun install"? error: Could not resolve: "ip-address". Maybe you need to "bun install"? at /opt/node_modules_cache/socks/build/client/socksclient.js:15 The cached node_modules layer in the pre-baked Docker image had `socks` (the new dep) but was missing its transitive deps (smart-buffer, ip-address). The image build copied only package.json into the build context — without bun.lock, `bun install` resolved a different tree than local `bun install` did, dropping required transitive deps. Reproduces locally as 229 packages (correct) when bun.lock is present or absent. Why CI diverged isn't fully understood — possibly Docker layer cache reuse across image rebuilds — but the deterministic fix is to include the lockfile in the image build context and use `--frozen-lockfile`, matching what every CI doc recommends. Changes: - .github/docker/Dockerfile.ci: COPY bun.lock alongside package.json, switch `bun install` → `bun install --frozen-lockfile` so any future lockfile drift fails loudly during image build instead of producing a partially-installed cache that breaks downstream eval jobs. - .github/workflows/evals.yml: include bun.lock in the image-tag hash so adding/removing a dep invalidates the image, AND copy bun.lock into the docker context alongside package.json. - .github/workflows/evals-periodic.yml: same updates. - .github/workflows/ci-image.yml: rebuild trigger now fires on bun.lock changes too; build context includes bun.lock. Image hash changes → fresh image gets built on next CI run → install matches the lockfile exactly → no missing transitive deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): use hardlink copy instead of symlink for node_modules cache After the bun.lock fix landed, the eval matrix STILL failed identically: Could not resolve: "smart-buffer" / "ip-address" at /opt/node_modules_cache/socks/build/client/socksclient.js But the hash-tagged image actually contains smart-buffer + ip-address + socks all flat in /opt/node_modules_cache (verified by pulling and inspecting the image). 207 packages, all present. Root cause: the workflow used `ln -s /opt/node_modules_cache node_modules` to restore deps. Bun build (and Node module resolution generally) walks a file's realpath to find sibling deps. From the symlinked /workspace/node_modules/socks/build/client/socksclient.js, realpath resolves to /opt/node_modules_cache/socks/build/client/socksclient.js, and walking up to find a node_modules/smart-buffer dir fails — there's no `node_modules` segment in the realpath. Switch `ln -s` → `cp -al` (hardlink-copy). Each file in the cache becomes a hardlink at /workspace/node_modules/<pkg>, sharing inodes (no data copy). Realpath of /workspace/node_modules/socks/.../socksclient.js stays inside /workspace/node_modules, so sibling deps resolve correctly. Speed is comparable to symlink — `cp -al` on ~200 packages on tmpfs is sub-second. Same caching story preserved. Both evals.yml and evals-periodic.yml updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): cp -r instead of cp -al — /opt and /workspace are different filesystems The hardlink-copy fix landed and immediately broke with: cp: cannot create hard link 'node_modules/<file>' to '/opt/node_modules_cache/<file>': Invalid cross-device link GitHub Actions runners mount the workspace volume at /workspace (overlay-fs layered onto the runner image), and /opt is the runner image's own filesystem. Cross-filesystem hardlinks aren't supported. Switch `cp -al` → `cp -r`. Cost: ~5s for ~200 packages of small JS files vs ~0s for the broken symlink. Still cheaper than the ~15s `bun install` fallback. Realpath of /workspace/node_modules/<pkg>/... stays inside /workspace, so bun build's sibling-dep resolution works. Both evals.yml and evals-periodic.yml updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

garrytan mentioned this pull request May 7, 2026

v1.28.0.0 feat: browse --headed/--proxy/--navigate + gstack/llms.txt + webdriver-only stealth #1363

Merged

6 tasks

garrytan closed this May 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(browse): add --navigate flag to download for browser-triggered file downloads#1355

feat(browse): add --navigate flag to download for browser-triggered file downloads#1355
garrytan-agents wants to merge 1 commit intogarrytan:mainfrom
garrytan-agents:feat/browse-download-handler

garrytan-agents commented May 7, 2026 •

edited by blacksmith-sh Bot

Loading

Uh oh!

garrytan commented May 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

garrytan-agents commented May 7, 2026 • edited by blacksmith-sh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Usage

Testing

Changes

Uh oh!

garrytan commented May 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

garrytan-agents commented May 7, 2026 •

edited by blacksmith-sh Bot

Loading