Skip to content

feat(browser): agent-native payload — network bodies, html tree budgets, extract command#1104

Merged
jackwener merged 2 commits intomainfrom
feat/browser-agent-native-payload
Apr 21, 2026
Merged

feat(browser): agent-native payload — network bodies, html tree budgets, extract command#1104
jackwener merged 2 commits intomainfrom
feat/browser-agent-native-payload

Conversation

@jackwener
Copy link
Copy Markdown
Owner

Why

Three independent agent-usage gaps in the browser surface, addressed as one complete change per direction from #opencli-browser thread:

  1. network --detail silently cut response bodies at 4000 chars, breaking JSON.parse on any non-trivial API response. Purely invisible to the agent.
  2. get html --as json had no budget knobs, so the agent either got a full-page tree dump or nothing — no middle ground when scoping down.
  3. No content-reading channel existed. Agents were misusing get html to try to read articles, which returns DOM structure, not prose.

All three are about the same thing from first principles: agent-readable payload shape, with visible truncation so the agent can continue rather than fail silently.

What changed

browser network — body truncation becomes visible

  • src/browser/cdp.ts + extension/src/cdp.ts: raise the silent 4KB slice to an 8MB memory-guard cap and record responseBodyFullSize / responseBodyTruncated (extension path also gets request-body mirrors).
  • src/browser/network-cache.ts: CachedNetworkEntry.bodyTruncated propagates through.
  • src/cli.ts browser network:
    • List view now emits body_truncated_count + per-entry body_truncated flag.
    • --detail envelope emits body_truncated, body_full_size, body_truncation_reason: 'capture-limit' | 'max-body'.
    • New --max-body <chars> flag for explicit caller-side cap (default 0 = unlimited).

browser get html --as json — budget knobs

  • src/browser/html-tree.ts: add depth / childrenMax / textMax budgets to the in-page serializer. The result envelope includes truncated: {depth, children_dropped, text_truncated} only when a budget was actually hit.
  • src/cli.ts: expose as --depth <n> / --children-max <n> / --text-max <n>.

browser extract — new agent-native reading channel

  • src/browser/extract.ts (new):
    • buildExtractHtmlJs(selector) — in-page clone + denoise (strip script/style/nav/header/footer/aside/iframe/form/button/... + role=navigation/banner/contentinfo/complementary + aria-hidden=true; scrub on*/style/data-* attrs).
    • htmlToMarkdown (existing helper) → structure-aware chunkMarkdown that snaps chunk end to the nearest \n\n (or \n) within a 15% boundary window, falling back to a hard cut.
    • Stateless resume via next_start_char cursor.
  • src/cli.ts: new browser extract [url] --selector <sel> --start <n> --chunk-size <n> command.

Truncation signal contract (all three channels)

Every channel makes cuts visible to the agent in the envelope:

Channel Signals
network list body_truncated_count, per-entry body_truncated
network detail body_truncated, body_full_size, body_truncation_reason
get html json truncated: { depth?, children_dropped?, text_truncated? } (only when hit)
extract total_chars, chunk_size, start, end, next_start_char

Test plan

  • pnpm test src/browser/extract.test.ts — 13 tests pass (chunking boundaries, resume chain, min-clamp, envelope wrapping)
  • pnpm test src/browser/html-tree.test.ts — all green incl. 5 new budget tests
  • pnpm test src/cli.test.ts — all green incl. 5 new body-truncation-signal tests
  • pnpm typecheck — clean
  • Manual smoke: browser network --detail ... --max-body 500 returns body_truncated: true + body_full_size
  • Manual smoke: browser extract <url> on a real article page + resume via --start <next>

Scope discipline

  • No new dependencies (turndown was already in package.json; htmlToMarkdown already exists in src/utils.ts).
  • No changes to list-view default behavior except the new body_truncated_count field (additive).
  • No backwards-incompatible changes to existing flags.

cc @codex-mini1 @First-principles-1 — requesting review per thread assignment.

…ts, extract command

Three fixes/additions driven by agent-usage gaps, as one complete change:

- network (P0 fix): lift silent 4000-char body truncation in CDP + extension
  paths to an 8MB memory-guard cap, and surface body_truncated / body_full_size
  / body_truncation_reason in the --detail envelope so the agent sees when a
  body was cut. List view also exposes body_truncated_count and per-entry flag.
  Adds --max-body flag for explicit caller-side capping.

- get html --as json (P1): add --depth / --children-max / --text-max budget
  knobs on the tree serializer, plus a truncated={depth,children_dropped,
  text_truncated} envelope that only appears when a budget is hit. Lets the
  agent narrow DOM output without walking away empty-handed.

- extract (P2 new command): agent-native article/content channel. Scope →
  denoise (strip nav/header/footer/scripts/forms/etc.) → HTML→markdown via
  existing htmlToMarkdown → paragraph-boundary-aware chunk with stateless
  next_start_char resume cursor. Agents no longer misuse `get html` to read.
…/fallback

Addresses review blockers on #1104:

- NETWORK_INTERCEPTOR_JS fallback no longer silently drops bodies above the
  per-entry cap. Raised cap to 1 MiB (ring stays at 200 entries), and on
  overflow keeps the string prefix + sets `bodyTruncated` / `bodyFullSize`
  so `browser network` propagates the same agent-visible signal the CDP /
  extension paths emit.

- `CachedNetworkEntry` schema switches from internal camelCase
  `bodyTruncated` to the user-facing `body_truncated` / `body_full_size`
  fields. `--raw` emits cache entries verbatim, so this removes the
  snake_case/camelCase split across list / --detail / --raw.

- Adds a `--raw` truncation-contract test that also asserts the camelCase
  fields do not leak through.
@jackwener jackwener merged commit acb08a4 into main Apr 21, 2026
14 checks passed
@jackwener jackwener deleted the feat/browser-agent-native-payload branch April 21, 2026 04:17
luxiaolei pushed a commit to luxiaolei/OpenCLI that referenced this pull request Apr 21, 2026
…ts, extract command (jackwener#1104)

* feat(browser): agent-native payload — network bodies, html tree budgets, extract command

Three fixes/additions driven by agent-usage gaps, as one complete change:

- network (P0 fix): lift silent 4000-char body truncation in CDP + extension
  paths to an 8MB memory-guard cap, and surface body_truncated / body_full_size
  / body_truncation_reason in the --detail envelope so the agent sees when a
  body was cut. List view also exposes body_truncated_count and per-entry flag.
  Adds --max-body flag for explicit caller-side capping.

- get html --as json (P1): add --depth / --children-max / --text-max budget
  knobs on the tree serializer, plus a truncated={depth,children_dropped,
  text_truncated} envelope that only appears when a budget is hit. Lets the
  agent narrow DOM output without walking away empty-handed.

- extract (P2 new command): agent-native article/content channel. Scope →
  denoise (strip nav/header/footer/scripts/forms/etc.) → HTML→markdown via
  existing htmlToMarkdown → paragraph-boundary-aware chunk with stateless
  next_start_char resume cursor. Agents no longer misuse `get html` to read.

* fix(browser): unify body-truncation signal contract across raw/detail/fallback

Addresses review blockers on jackwener#1104:

- NETWORK_INTERCEPTOR_JS fallback no longer silently drops bodies above the
  per-entry cap. Raised cap to 1 MiB (ring stays at 200 entries), and on
  overflow keeps the string prefix + sets `bodyTruncated` / `bodyFullSize`
  so `browser network` propagates the same agent-visible signal the CDP /
  extension paths emit.

- `CachedNetworkEntry` schema switches from internal camelCase
  `bodyTruncated` to the user-facing `body_truncated` / `body_full_size`
  fields. `--raw` emits cache entries verbatim, so this removes the
  snake_case/camelCase split across list / --detail / --raw.

- Adds a `--raw` truncation-contract test that also asserts the camelCase
  fields do not leak through.

(cherry picked from commit acb08a4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant