Skip to content

feat: agent-native retrospective — analyze / verify guards / fixture content checks#1133

Merged
jackwener merged 5 commits intomainfrom
feat/agent-native-retrospective
Apr 21, 2026
Merged

feat: agent-native retrospective — analyze / verify guards / fixture content checks#1133
jackwener merged 5 commits intomainfrom
feat/agent-native-retrospective

Conversation

@jackwener
Copy link
Copy Markdown
Owner

Summary

Post-mortem on slow 1point3acres + 51job adapter sessions, consolidated into one PR. Scope = "reduce uncertainty and catch silent failures" — the two things that sink agent success rate on first-time adapters.

Principle guarding scope: agents do NOT fear repetition/boilerplate/handwriting; agents DO fear uncertainty (which path / which request / what to persist) and silent failures (verify-green but data is wrong). Every change here maps to one of those two.

What changed

New commands

  • browser analyze <url> — one-shot site recon. Returns pattern (A/B/C/D classification), anti_bot (Aliyun WAF / Cloudflare / Akamai / Geetest vendor detection + evidence), nearest_adapter (registry lookup for adapter reuse), and a single-sentence recommended_next_step. Replaces the manual open → wait → network → squint → guess loop.
  • browser wait xhr <regex> — poll for a specific XHR URL. Replaces blind wait time 5 on SPA pages; data-arrival barriers become deterministic.

New fixture rules (catch silent failures)

  • mustNotContain: { col: [substrs] } — catches content contamination (sibling DOM bleed, breadcrumb prefixes, e.g. description accidentally containing address: / category: from a tag node). notEmpty does not catch this.
  • mustBeTruthy: [col] — catches || 0 / || false silent fallbacks on numeric/boolean columns. notEmpty / types do not catch this.

Guards

  • verify post-success site-memory check + --strict-memory flag. Verify-green now surfaces when ~/.opencli/sites/<site>/ was never written back; --strict-memory makes the missing memory a hard failure. Memory only materializes if authors write it; previously there was no feedback loop.
  • CI: cli-manifest.json drift guard. Committed manifest must match a fresh npm run build. Main was already drifted (PR fix(bilibili): resolve full video URLs and preserve full description #1118 left bilibili/video reordering + a jianyu since_days arg not committed to the manifest); this PR regenerates it and the guard prevents the next drift.

Docs (opencli-adapter-author + opencli-autofix)

  • New: references/success-rate-pitfalls.md — 10 concrete silent-failure scenarios from real sessions, each with defense via fixture/adapter patterns.
  • autofix: Discipline rule fix: escape evaluate args in browser scripts #6 — verify pattern failure = tighten adapter, never loosen the fixture. Fixture relaxation is the canonical way to silently accept broken data.
  • site-recon.md: Leads with browser analyze; manual 3-step is the fallback. wait time 3wait time 2 (2s is enough; wait xhr is more robust still).
  • api-discovery.md: New §0 covering WAF vendor detection and cross-subdomain CORS — the two gotchas that burned the 51job session (acw_sc__v2 cookie meant the Node fetch was always going to return slider HTML; cupid.51job.com from jobs.51job.com was CORS-blocked even with credentials:'include').
  • site-memory.md: Documents the new fixture fields; adds explicit warning not to loosen patterns to pass verify.

Test plan

  • npx tsc --noEmit clean
  • npx vitest run src/ — 845 passed / 2 skipped
  • npm run build idempotent (only drift is from main's preexisting stale manifest — which this PR fixes)
  • New unit tests: 16 for analyze.ts (WAF detection, pattern classification, nearest-adapter matching, top-level assembly) + 3 for fixture content rules
  • Dogfood browser analyze https://www.51job.com/ after merge — expects anti_bot.vendor = aliyun_waf
  • Dogfood browser wait xhr '/api/...' on a slow SPA after merge

Review focus

Two pairs of reviewers worth: agent-UX (skill docs + analyze output shape) and CLI correctness (wait xhr polling, verify memory-write check, manifest drift guard).

Background: one PR consolidation explicitly approved by WAWQAQ (msg 0de55bc8, "有必要的我们都做,全部一次性做掉").

…content checks

Post-mortem on slow 1point3acres + 51job adapter sessions, consolidated
into one PR. Scope is "reduce uncertainty and catch silent failures"
— the two things that sink agent success rate on first-time adapters.

Changes:
- `browser analyze <url>` — one command returns pattern (A/B/C/D),
  anti-bot vendor (Aliyun/Cloudflare/Akamai/Geetest), nearest adapter,
  and a single-sentence recommended_next_step. Replaces the three-step
  open/wait/network recon loop when it can reach a confident verdict.
- `browser wait xhr <regex>` — poll for a specific XHR URL instead of
  blind `wait time N`, so SPA data-arrival barriers are deterministic.
- Fixture `mustNotContain` / `mustBeTruthy` — catch two silent-failure
  modes `notEmpty` misses: content contamination (sibling DOM bleed)
  and `|| 0` / `|| false` fallbacks.
- `browser verify` post-success site-memory check + `--strict-memory`
  — verify-green no longer hides the case where `~/.opencli/sites/`
  was never written back. Memory only materializes if authors write it.
- CI: guard that committed `cli-manifest.json` matches a fresh build.
  Main was already drifted (#1118 left stale ordering + a missing arg);
  this PR regenerates the manifest and will catch the next drift.

Docs (opencli-adapter-author + opencli-autofix skills):
- `success-rate-pitfalls.md` — 10 concrete silent-failure scenarios
  seen in real adapter sessions, each with defense via fixture /
  adapter patterns.
- `autofix` gains discipline rule #6: verify pattern failure means
  tighten the adapter, never loosen the fixture.
- `site-recon.md` leads with `browser analyze`; `api-discovery.md`
  adds a §0 covering WAF vendor detection and cross-subdomain CORS
  (the two gotchas that burned the 51job session).
- `wait time 3` → `wait time 2`, with `wait xhr` as the robust choice.
Three adapters (chatgpt/image, gemini/image, instagram/download) baked
`path.join(os.homedir(), ...)` into the `default` field of their args.
The committed manifest therefore carried my personal `/Users/jakevin/...`
paths — which agents running on a different host saw as surprising
defaults. The drift guard I just added to CI caught it on the first run.

Runtime behavior is unchanged: each adapter still falls back to
`path.join(os.homedir(), …)` inside `func` when the kwarg is absent.
Only the displayed / registered default becomes a tilde-path.
@jackwener jackwener merged commit dc72426 into main Apr 21, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant