openrouter scraper: SelectorTimeout at signup_flow — DOM drift

[hanwencheng]

Symptom

agentkeys --session-id alice provision openrouter against live broker.litentry.org consistently fails at the openrouter signup step with a SelectorTimeout tripwire.

{"level":"info","event":"provision_metric","name":"trip_wire_fired","service":"openrouter","kind":"SelectorTimeout","step":"signup_flow"}
Problem: A script step timed out at 'signup_flow'.
Cause: The target site's DOM may have changed (tripwire: SelectorTimeout).

Scope (what's broken vs what works)

• ✅ Auto-provision pipeline works end-to-end: agentkeys provision loads the saved session JWT, calls /v1/mint-oidc-jwt, exchanges via AssumeRoleWithWebIdentity, injects AWS temp creds into the scraper subprocess. The fact that trip_wire_fired fires AT ALL is proof of life that the pipeline ran.
• ❌ openrouter's signup-page DOM has drifted since the scraper's selectors were last verified.

Affected file

provisioner-scripts/src/scrapers/openrouter.ts — the signupEmailOtp pattern's selectors no longer match openrouter's current signup flow.

Current hardcoded selectors at the top of the file:

• EMAIL_SELECTOR = 'input[name="email"]'
• SUBMIT_BUTTON_SELECTOR = 'button[type="submit"]'
• OTP_SELECTOR = 'input[name="otp"]'
• VERIFY_BUTTON_SELECTOR = 'button[type="submit"]'

Proposed systemic fix

See comment #issuecomment-4455841217 for the 4-layer scalable selector-healing plan covering ALL provisioner scrapers (forensics → fallback chains → nightly canary → LLM auto-heal). Supersedes the per-incident "Fix path" below.

Fix path (per-incident, pre-systemic-fix)

Run the workflow recorder against the current openrouter signup page:

# Dev-time
# See SKILL: agentkeys-workflow-collection / agentkeys-record-scraper

…then update the selectors (or migrate to the *-cdp.ts variant if openrouter has anti-bot defenses that block Playwright's default Chromium).

Demo-doc note

docs/stage7-demo-and-verification.md §5.3 now explicitly states that a trip_wire_fired log line proves the auto-provision pipeline succeeded — scraper maintenance is out of scope for that demo. So this issue doesn't gate the §5.3 demo claim; it's a follow-up to keep openrouter usable as an example service.

Repro

bash scripts/agentkeys-init-email-demo.sh --session-id alice
set -a; source scripts/operator-workstation.env; set +a
export AGENTKEYS_BROKER_URL=https://broker.litentry.org
export AGENTKEYS_DATA_ROLE_ARN=arn:aws:iam::${ACCOUNT_ID}:role/agentkeys-data-role
export AWS_REGION=us-east-1
export AGENTKEYS_SIGNER_URL=$BACKEND_URL
export AGENTKEYS_SESSION_ID=alice
(cd provisioner-scripts && npm install && npx playwright install chromium)
agentkeys --session-id alice provision openrouter

Last verified failing: 2026-05-14.

[hanwencheng]

Issue #83 — Scalable selector-healing for provisioner scrapers

Issue: #83 openrouter scraper SelectorTimeout — DOM drift (label: provision-fix)

Scope of this plan: make DOM drift a self-healing class of failure for ALL provisioner scrapers, not a per-service firefight. openrouter is the first known case; brave / exa / groq / anthropic (planned services) will hit the same trap.

Plan-status: awaiting user approval before any implementation. Do not start coding until the user signs off.

---

Problem statement

Every scraper under provisioner-scripts/src/scrapers/ hardcodes a fixed set of CSS selectors (e.g. input[name="email"]). When the target site changes its DOM:

1. The scraper silently breaks until someone runs agentkeys provision <service>.
2. The tripwire fires with kind:"SelectorTimeout" + step:"signup_flow" — useful for "what broke" but lacks forensics: no DOM snapshot, no screenshot, no candidate-selector suggestions. The operator has to re-run the workflow-collection skill (~5–10 min of human-supervised browser driving) just to see what the new DOM looks like.
3. No early-warning system. Drift is only caught when an operator attempts to provision.
4. Fix-cycle is manual: re-record → diff old vs new selectors → edit scraper → test → commit. ~30 min minimum per service.

This will compound badly. Stage 5 plans 8+ services (development-stages.md); at industry-typical DOM-change cadence (~1 change every 2–3 months per service), we'd expect 3–4 broken scrapers at any given time once all services land.

---

What we already have (don't rebuild this)

Asset	Location	What it does
Workflow recorder	provisioner-scripts/src/workflow-recorder/	Drives a real browser, captures per-step screenshots + DOM snapshots + digest + Playwright trace. Emits draft-scraper.ts skeleton with detected regexes.
Pattern composition	provisioner-scripts/src/patterns/signup_email_otp.ts	Reusable signup → email-OTP → key-mint flow. Scrapers compose patterns + provide selectors. Selectors are the only service-specific surface.
CDP variant	*-cdp.ts (openrouter, openai)	Connects to a user-launched real Chrome via CDP, avoids --enable-automation flag that Cloudflare Turnstile blocks. Production path.
Tripwire event	provisioner-scripts/src/types.ts line 20	{type:"tripwire", kind, step, elapsed_ms} — emitted on selector timeouts. Consumed by daemon's spawn_and_collect.
Two dev-time skills	~/.claude/skills/agentkeys-{workflow-collection,record-scraper}	Human-supervised recording (slow but reliable). Trigger: "add a new provisioner" / "record a scraper".

The healing plan leverages all of this — doesn't replace any of it. It adds 4 layers on top that turn manual fix-cycle into automated detect-+-propose-fix.

---

The plan — 4 layers, lowest-effort first

Layer 1 — Rich tripwire forensics (1 day, foundation)

Goal: every tripwire becomes self-describing. A future session (human or LLM) reading the tripwire alone can propose a fix without re-running the live demo.

Changes:

• Extend TripwireEvent in types.ts:

  { type:"tripwire", kind, step, elapsed_ms,
    page_url:string, dom_digest_path:string, screenshot_path:string,
    expected_selector:string, candidate_selectors:string[] }

• Wrap every page.waitForSelector call in lib/playwright-patterns.ts with a waitForSelectorOrCapture(page, selector, opts) helper that on timeout:
  1. Saves page.content() to ~/.agentkeys/provision-tripwires/<service>/<ts>/dom.html (full DOM)
  2. Computes a Claude-friendly digest (visible-text + form structure, ~5KB) → digest.md
  3. Saves page.screenshot() → screenshot.png
  4. Heuristic-extracts candidate selectors from the DOM for the failed step's role (e.g. for "email input" → all input[type="email"], input[name*="email"], [autocomplete*="email"] elements)
  5. Emits the rich tripwire event, then re-throws
• Where the heuristics live: lib/selector-heuristics.ts (new file). Map step name → element-role → candidate-extraction strategy.

Outcome: the ~/.agentkeys/provision-tripwires/ directory becomes a forensics archive. provision-fix issues can be auto-filed with screenshots + DOM links inline.

Layer 2 — Selector fallback chains (1–2 days)

Goal: a single DOM change shouldn't break a scraper. Try N candidates, fall through.

Changes:

• Patterns (signup_email_otp.ts) accept string | string[] for every selector field.
• New helper waitForFirstSelector(page, candidates, opts) — races each candidate, returns the first that lands within timeout. Tripwires only when ALL fail.
• Scrapers change from:

  const EMAIL_SELECTOR = 'input[name="email"]';

  to:

  const EMAIL_SELECTORS = [
    'input[name="email"]',          // primary (current canonical)
    'input[type="email"]',          // generic fallback
    'input[autocomplete="email"]',  // accessibility fallback
    '[data-testid*="email"]',       // testid fallback
  ];

• Convention: list selectors most-specific-first. Generic fallbacks at the end. Comments document each entry's last-verified date.

Outcome: roughly half of DOM changes (renamed name attribute, swapped id, added data-testid instead of removing the old one) heal without code changes. The fallback chains absorb the drift.

Layer 3 — Nightly canary in CI (1 day)

Goal: catch drift in CI before any operator attempts provision. The canary runs against live sites; production-impacting drift never reaches operators.

Changes:

• New file provisioner-scripts/scripts/canary-run.ts:
  • For each scraper in src/scrapers/*.ts (skip *-cdp.ts — those need real Chrome, run separately):
    • Spawn a fresh Playwright Chromium
    • Run the scraper's signupFlow() to the point of first email submission (do NOT submit the email — just verify selectors all match)
    • Record {service, status: "passing"|"tripwire", tripwire_path?} to canary-report.json
• New GitHub Action .github/workflows/scraper-canary.yml:
  • Schedule: cron: "0 3 * * *" (daily at 3 UTC)
  • Runs npm run canary in provisioner-scripts/
  • On any tripwire: auto-open / auto-update a provision-fix issue per affected service. Title: <service>: SelectorTimeout at <step> — DOM drift detected by canary <date>. Attach the rich-tripwire forensics (Layer 1 outputs) as issue comment.
  • Dedup: if a provision-fix issue for that service is already open, append a comment instead of opening a duplicate.

Outcome: drift is detected within 24h of the target site changing. The forensics blob lands in the GitHub issue automatically, ready for Layer 4 (or a human) to act on.

Layer 4 — LLM-driven auto-heal (2–3 days)

Goal: when Layer 3 detects drift + Layer 1 captured forensics, attempt to heal automatically. Human review only on auto-heal failure.

New skill agentkeys-heal-scraper (lives at ~/.claude/skills/agentkeys-heal-scraper/):

• Triggered manually OR by the canary workflow when a tripwire fires
• Args: <service> (e.g. openrouter)
• Reads the latest ~/.agentkeys/provision-tripwires/<service>/<ts>/ blob
• Feeds DOM digest + screenshot + step name + expected element role to Claude with a prompt template like:

  "The scraper for <service> timed out waiting for <expected_selector> at step <step>. Page URL: <url>. Here's the DOM digest and screenshot. Propose 3 CSS selectors that target the same element on this page, ranked most-specific to most-generic. Output as a JSON array."

• Each candidate runs against a fresh Playwright session (via the canary's harness)
  • If any candidate lets the scraper progress to the next step, it's added to the fallback chain
  • Picks the highest-specificity candidate that worked
• On success:
  • Edits the scraper file: appends new selector to the head of the fallback chain with a // last-verified: <date> comment
  • Opens a PR with the change + before/after screenshots
  • Comments on the GitHub issue: "Auto-heal proposed — see PR #N"
• On failure (no candidate worked):
  • Comments on the issue: "Auto-heal failed — flow may need re-recording. Run /agentkeys-workflow-collection to capture the new flow."

Outcome: the median DOM-drift incident heals in CI without human touch. The minority where the DOM changed structurally (e.g. signup became a multi-step wizard) fall back to the existing workflow-recorder skill — same path operators use today, but only invoked when auto-heal can't recover.

---

Phasing recommendation

Phase	Lands	Acceptance
Layer 1	Forensics infrastructure	A fresh agentkeys provision openrouter run produces ~/.agentkeys/provision-tripwires/openrouter/<ts>/{dom.html, digest.md, screenshot.png, tripwire.json}
Layer 2	Fallback chains	openrouter scraper uses EMAIL_SELECTORS: string[] with 4 candidates; a manually-broken primary selector triggers fallback to generic, scraper still passes
Layer 3	Nightly canary	First nightly run files a provision-fix issue for openrouter (since it's known broken); subsequent runs dedup
Layer 4	Auto-heal skill	Manually run agentkeys-heal-scraper openrouter produces a PR with updated selectors; CI canary passes after merge

Ship-order rationale: Layer 1 is a prereq for Layers 3 + 4. Layer 2 stands alone and immediately reduces drift incidents. Layers 3 + 4 together close the auto-heal loop. Each layer is independently shippable; partial deployment still delivers value.

---

Scope & non-goals

In scope:

• All scrapers under provisioner-scripts/src/scrapers/*.ts (currently openrouter, openrouter-cdp, openai-cdp; future: brave / exa / groq / anthropic)
• Selector drift only (CSS attribute changes, renamed name / id / data-testid)

Out of scope:

• Anti-bot / CAPTCHA defenses (hCaptcha, reCAPTCHA, Turnstile, proof-of-work) — separate problem class; the CDP variant already mitigates some of this
• Multi-step wizard restructures (signup-as-3-pages → signup-as-1-page) — Layer 4 escalates these to the workflow-recorder skill
• Login-flow drift (currently we only do signup → key-mint; login is unused)
• Per-service email-provider changes (handled by lib/email.ts + lib/email-backends)

---

Open questions for review

1. Should Layer 3's canary actually submit emails to real services? Pro: full end-to-end coverage. Con: creates dummy accounts at the target's expense, may trigger anti-abuse. Recommendation: NO — stop the canary just before email submission (verify all pre-submit selectors exist + are interactable, then bail). The "key-mint after verify" path stays manually validated.
2. Where do Layer 1 forensics live in CI? Local dev: ~/.agentkeys/provision-tripwires/. CI: GitHub Actions artifact (retention 30 days) + the GitHub issue body link. Need to decide if we also mirror to S3 for long-term archive.
3. Layer 4 auto-heal: does the PR auto-merge after CI passes, or require human approval? Recommendation: human approval for now (selector changes are low-risk but not zero-risk). Re-evaluate after 10 successful auto-heal PRs land.
4. Should we backfill fallback chains for openrouter-cdp.ts and openai-cdp.ts as part of Layer 2? Recommendation: YES — same blast-radius reduction applies. Both CDP scrapers benefit equally.

---

Estimate

• Layer 1: 1 day (forensics helper + types + heuristic file)
• Layer 2: 1–2 days (helper + pattern signature change + per-scraper migration)
• Layer 3: 1 day (canary script + GHA workflow)
• Layer 4: 2–3 days (new skill + Claude prompt template + CI integration)

Total: 5–7 dev-days to land all four layers. Layer 1 + 2 alone (3 days) already cuts incident rate in half.

---

Success metrics (post-landing)

• Mean time to detect DOM drift: drops from "operator-driven, indefinite" → "<24h via nightly canary"
• Mean time to fix DOM drift: drops from "~30 min human review" → "<5 min for auto-healable cases, ~30 min only for structural changes"
• % of drift incidents auto-healed: target 60% after Layer 4 lands (selector renames / attribute swaps); remaining 40% (structural changes) re-recorded via existing workflow-recorder skill
• Cumulative scraper-broken-time across N services: linear-in-N today; sublinear (mostly constant) after this lands