feat(stage6): SES-S3 email + workflow-recorder skill + OpenRouter/OpenAI production scrapers#52
Merged
hanwencheng merged 29 commits intomainfrom Apr 23, 2026
Merged
Conversation
The Gmail-setup section ended at 'Daemon running and paired — see the Stage 4 manual test guide', but Stage 5a provision doesn't actually need a paired daemon: cmd_provision runs as the master CLI and uses session.wallet as the agent_id. A reader who finished Gmail setup had no concrete path from there to a running demo. Replace the vague pointer with an explicit two-terminal runbook: - Terminal 1: mock backend (Stage 5a stores into the mock; real Heima lands in v0.1). - Terminal 2: agentkeys init --mock-token + agentkeys provision openrouter + verification (read back the key, curl OpenRouter). Plus the 'under the hood' breakdown so a reader knows why no daemon or pairing is involved, and a short 'artifacts to inspect' pointer (session.json path, audit JSONL). Also promotes the build-and-install step from prose to step 5 so the prerequisites list is self-contained and paste-able.
Three tightly-coupled changes so the Stage 5a live demo is both re-runnable (returning-user collision) and debuggable (no more silent 'subprocess ended without terminal event' with no cause). provisioner-scripts/src/scrapers/openrouter.ts - Split AGENTKEYS_EMAIL_USER (canonical IMAP login) from a new AGENTKEYS_SIGNUP_EMAIL (what we type into OpenRouter's signup form). Gmail IMAP rejects plus-addressing at login, so the two had to diverge before plus-addressing could work at all. - Wrap main() in a catch-all that emits a terminal error event and flushes stdout before process.exit. Playwright launch failures, dynamic-import errors, IMAP connection refusals, and any other throws upstream of the scraper's inner try/catch now surface as a parseable Error event instead of dying silently and being reported as 'subprocess ended without terminal event.' crates/agentkeys-provisioner/src/orchestrator.rs - On the no-terminal-event error path, best-effort-write the full subprocess output (exit code, every event emitted, complete stderr) to ~/.agentkeys/logs/provision-<service>-<ts>.log and include the path in the error message. stderr_tail (20 lines) stays inline for the quick case. docs/manual-test-stage5.md - Flip the primary demo path from 'dedicated throwaway Gmail' to 'your existing Gmail + plus-addressing + app password.' Reason documented: OpenRouter's /auth is signup+signin on one URL, so reusing a canonical address across runs always fails on the second run with a returning-user UI the scraper wasn't designed for. Plus-addressing minted per-run via $(date +%s) gives us DWD-equivalent disposable emails at zero infrastructure cost. - Document the two env vars and why they exist separately. - Dedicated-throwaway-Gmail + Workspace DWD demoted to <details> alternatives. - New 'Debugging a failure' block under Artifacts pointing to the persistent log file + the direct-scraper-run fallback. - New 'subprocess ended without terminal event' and 'account already exists (returning-user path)' entries in Failure modes. Tests: - cargo test -p agentkeys-provisioner --release: 15/15 pass - npm test --prefix provisioner-scripts: 15/15 pass across 6 files
Root cause of the 'exit_code: Some(0) / events_emitted: 0 / stderr empty' failure mode: openrouter.ts declares `export default async function main()` but nothing at module scope invokes it. When the provisioner runs `npx tsx provisioner-scripts/src/scrapers/openrouter.ts`, the module loads (imports + constant declarations + function decls), reaches EOF, and exits cleanly without ever calling main(). The orchestrator then correctly reports 'no terminal event' because the scraper genuinely emitted none. Tests did not catch this because they only import the named export `runOpenRouterScraper`, not the default `main`. Add the standard Node ESM entry-point guard at the bottom of the file. main() runs only when the file is the direct script target (argv[1] matches import.meta.url). Named-export imports from test files still bypass it, so the 15/15 TS test suite stays green. Tests: - npx tsc --noEmit: clean - npm test --prefix provisioner-scripts: 15/15 pass across 6 files
harness/stage-5a-live-demo-handoff.sh: preflights the Stage 5a live demo end-to-end in a single bash run. Checks: - all 5 AGENTKEYS_EMAIL_* env vars present (fail-fast via :? with pointed error text for each) - target/release/agentkeys exists + executable - mock-server reachable at $BACKEND - node + npx on PATH - provisioner-scripts deps installed - Playwright chromium_headless_shell-* installed under $HOME (guards against the sandbox-HOME gotcha discovered in this ralph session — Playwright caches browsers per-HOME and a fresh HOME without cached browsers fails with "browserType.launch: Executable doesn't exist") Auto-mints AGENTKEYS_SIGNUP_EMAIL as <local>+or-<ts>@<domain> if unset so each run hits the OpenRouter signup path with a fresh email — no manual rotation needed. Executes the four Stage 5a acceptance criteria in order: 1. agentkeys init + provision openrouter (exit 0 required) 2. masked-key form check on stdout 3. agentkeys read openrouter returns sk-or-v1-... prefix 4. curl OpenRouter /api/v1/models returns HTTP 200 On failure, dumps the most-recent provision-openrouter-*.log so the user has the full stderr/events from the subprocess.
Three artifacts captured during ralph session driving the live OpenRouter provision to ground truth. harness/stage-5a-live-demo-handoff.sh: strip any existing plus-alias from AGENTKEYS_EMAIL_USER before appending +or-<ts>. Some email validators (including the one OpenRouter currently uses) reject double-plus addresses like agent+2026042001+or-...@wildmeta.ai and silently drop the signup. Gmail's inbound delivery path handles it fine; the signup form does not. provisioner-scripts/diag-imap.mjs: standalone probe that verifies IMAP auth works with the configured AGENTKEYS_EMAIL_* env, lists all mailboxes, and searches INBOX / Spam / All Mail / Trash for recent OpenRouter verification emails. Distinguishes "auth failed" / "email went to spam" / "email never arrived" failure modes that the scraper's EmailTimeout tripwire conflates. provisioner-scripts/diag-openrouter.mjs: standalone Playwright probe against the live openrouter.ai signup page. Captures screenshots + HTML snapshots + a JSON inventory of all input/button candidates to reveal where real DOM diverges from the scraper's hardcoded selectors. Used in this session to confirm OpenRouter migrated to Clerk (field name changed email -> emailAddress, button has no type=submit) — a Stage 5b blocker, not a Stage 5a bug.
harness/stage-5a-live-demo-handoff.sh - Drop misleading "JSON summary" claim from header — script prints SUCCESS but not JSON - Drop dead repo-root node_modules branch (never exists in this project; deps only live at provisioner-scripts/node_modules) - Collapse redundant step 4 header that had no check into step's 4 section (AC#1-#3 read-back check); renumber step 5 accordingly. Prior numbering was 1→2→3→4(empty)→5→6 with the 4th being just a comment. provisioner-scripts/diag-imap.mjs - Fix stale usage comment: file was moved from harness/ into provisioner-scripts/ (imapflow resolution) but the header still pointed at the old path. provisioner-scripts/diag-openrouter.mjs - Drop dead `|| candidates.find(...)` fallback in submit-button lookup. `buttons` is already filtered with the same /sign|continue|next|submit|start/i regex, so the fallback is a strict subset of the main filter and can never fire with a different value. Post-deslop regression: - cargo test --release -p agentkeys-provisioner: 15/15 pass - npm test --prefix provisioner-scripts: 15/15 pass across 6 files - handoff preflight smoke with no env: exit 1, clear missing-var msg
Stage 5b MVP CDP-connected scraper proven end-to-end, blocked on
email-duplicate. Pivot unblocked by adding throwaway-inbox
provisioning as a named Stage 6 deliverable.
provisioner-scripts/src/scrapers/openrouter-cdp.ts (new)
Connects to a user-launched real Chrome via chromium.connectOverCDP,
drives OpenRouter's Clerk-hosted signup form, polls Gmail IMAP for
the OTP, mints a key on /keys, prints sk-or-v1-* on stdout. Two
bugs fixed during the session:
- Click the checkbox INPUT directly, not the label (label wraps a
"Terms of Service" link that navigates to /terms)
- When the 180s Turnstile wait expires and URL is still /sign-up
with no OTP input present, fail explicitly instead of falling
through to a bogus OTP-waiting step.
Why CDP and not Playwright-launched Chromium:
Playwright's bundled Chromium ships with --enable-automation.
Cloudflare Turnstile detects this (error 600010) and refuses
to issue a token even when a human clicks the checkbox.
Connect to a real Chrome (launched with --remote-debugging-port)
bypasses this because the browser process has no automation
flags. Verified 2026-04-20: Turnstile passes invisibly in real
Chrome, Clerk backend returns clean responses.
Known blocker:
OpenRouter's Clerk integration normalizes Gmail/Workspace
plus-aliases to canonical. If agent@wildmeta.ai already has an
OpenRouter account, every plus-aliased variant gets rejected
with "email already in use." Only distinct local-parts work.
That's why Stage 6 throwaway inbox provisioning (bot-<id>@
agentkeys-email.io per call) is what unblocks the live demo.
provisioner-scripts/diag-or-{flow,turnstile,signin}.mjs (new)
Standalone Node probes used to diagnose the Turnstile failure.
Kept as runtime evidence for the Clerk-moved-to-Radix-UI
discovery and for future scraper authors' reference.
docs/manual-test-stage5.md (modified)
Section 4 rewritten from "when Stage 5b lands, future" to "CDP
scraper partial: proven working, blocked on email duplicate."
Includes: the run-recipe with Chrome --remote-debugging-port
command, required env, known blocker, Stage-6-dependent pickup
checklist.
docs/spec/plans/development-stages.md (modified)
Stage 6 deliverables extended with two named items:
- Throwaway inbox provisioning API: mint unique local-parts per
call (Clerk-normalization-proof), readable via the same
fetchVerificationCode shape the Stage 5b scraper uses.
- Stage 5b live-demo re-run: once throwaway provisioning lands,
re-run the CDP scraper end-to-end. Closes the manual-test-stage5
§4 pickup item.
Plus two test rows: email::throwaway_inbox_provisioning and
email::stage5b_live_demo_rerun.
docs/manual-test-stage6.md (new)
Stage 6 manual demo guide: preflight, provision-throwaway-inbox
walkthrough, per-user isolation test, Stage 5b live-demo re-run
procedure. Structured like Stage 5 doc so both are readable in
parallel.
.gitignore (modified)
Add .gstack/ — gstack creates .gstack/browse.json at repo root
during connect-chrome; not a repo artifact.
Post-change regression (fresh):
- cargo test --release -p agentkeys-provisioner: 15/15 pass
- npm test --prefix provisioner-scripts: 15/15 pass across 6 files
- Update how-to-use block to warn about Clerk's plus-alias normalization (SIGNUP_EMAIL must be a local-part OpenRouter hasn't seen) - Fix outdated '120s' claim in header — actual wait is 180s - Trim redundant log line that duplicated the block comment below it Post-deslop regression: - npm test --prefix provisioner-scripts: 15/15 pass - npx tsc --noEmit: clean
docs/stage6-aws-setup.md — copy-pasteable aws CLI sequence to go
from empty AWS account → live @agentkeys-email.io SES stack ready
for Stage 6 code to consume.
Sections:
1. Preconditions (IAM admin, aws CLI v2, domain decision)
2. Domain registration in Route 53 (+ alt-domain path)
3. SES domain identity + AWS-managed DKIM (interim; TEE-BYODKIM
flagged as post-heima-gaps §4 follow-up)
4. S3 bucket agentkeys-mail with per-user-isolation bucket policy
keyed off ${aws:PrincipalTag/agentkeys_user_wallet}
5. IAM role agentkeys-agent with two trust-policy variants:
4a OIDC-federated (preferred, needs §5 OIDC stub)
4b static IAM user (interim, OIDC deferred)
6. IAM OIDC provider registration (optional for interim)
7. SES receipt rule for inbound mail → S3
8. End-to-end test (send + read)
9. Hand-back checklist: exact outputs user shares back to wire
AGENTKEYS_EMAIL_BACKEND=ses-s3 in provisioner-scripts
10. Cleanup/teardown recipe
Each step has: aws CLI commands (not Console), explicit
<placeholder> markers, interim-vs-final flags, cross-references
to wiki/tag-based-access.md + heima-gaps-vs-desired-architecture.md
…org) User already has litentry.org in Route 53 (zone Z09723983CFJOHAE3VC65) on account 429071895007. The original runbook assumed a fresh agentkeys-email.io registration; reality is a subdomain carve-out is the faster + cheaper Stage 6 interim path. Changes: - §0 Preconditions: new "Domain decision" block comparing the subdomain path (default) vs the standalone agentkeys-email.io canonical path (post-interim). Env var defaults flipped to bots.litentry.org + PARENT_ZONE_ID=Z09723983CFJOHAE3VC65 + ACCOUNT_ID=429071895007 + BUCKET=agentkeys-mail-$ACCOUNT_ID. - §1 DNS: replaced "register a new domain" with "UPSERT records on the existing parent zone". Removed all route53domains / register-domain commands. - §2 SES verification: DNS heredoc now templates $DOMAIN and reads DKIM tokens into $T1/$T2/$T3 env so nobody hand-edits <tokenN> placeholders. - §2 DMARC: added note that rua=dmarc@$DOMAIN assumes the receipt rule in §6 is live; mailbox existence is a Stage 6 production follow-up. - §8 hand-back checklist: values pre-filled with the actual account/zone/domain (429071895007, Z09723983CFJOHAE3VC65, bots.litentry.org). Preserves the interim-vs-static IAM trust choice. - Interim DKIM-path note: clarified dkim/<domain>/v1 with explicit examples for both domain shapes. - .gitignore: add AWSCLIV2.pkg (user's aws CLI installer artifact) No semantics change to §3-§7 (bucket policy, IAM role, OIDC provider registration, receipt rule, test send) — those already flowed off $DOMAIN + $PARENT_ZONE_ID variables.
Two independent pieces landed together so PR-review sees them
as one Stage 6 step.
1. Mock-server inbox endpoints (US-6-2)
- SQLite schema: inboxes(address, agent_wallet, created_at)
+ inbox_messages(msg_id, address, from, subject, body,
received_at) with an idx on (address, received_at DESC).
- Handlers in crates/agentkeys-mock-server/src/handlers/inbox.rs:
* POST /mock/inbox/provision — bearer auth + ownership check;
returns bot-<hex>@<AGENTKEYS_EMAIL_DOMAIN>.
* POST /mock/inbox/deliver — unauthed test hook that simulates
inbound SES delivery; writes a row to inbox_messages.
* GET /mock/inbox/messages — bearer auth + ownership check on
the agent_wallet resolved from the address; returns messages
newest-first.
- Domain is configurable via AGENTKEYS_EMAIL_DOMAIN env
(default agentkeys-email.io, override to bots.litentry.org
for Stage 6 interim).
- 56/56 mock-server tests pass (existing 53 + 3 new: unique
addresses, deliver+fetch roundtrip, cross-session list denied).
2. OIDC interim stub service (US-6-5)
- services/oidc-stub/: express + jose ES256 service serving
/.well-known/openid-configuration, /.well-known/jwks.json,
and dev-only POST /internal/sign.
- Keys: generated at first run, persisted at
~/.agentkeys/oidc-stub/keypair.json (mode 0600). KMS path
reserved as a TODO branch gated on AGENTKEYS_OIDC_KMS_KEY_ID.
- README labels the service a TEE-interim stub with a clear
follow-up path: replace signer with TEE-derive(oidc/issuer/v1)
once heima-gaps §3 lands.
- 9/9 vitest tests pass: discovery doc shape, JWKS format,
private key never leaks, end-to-end JWT signs + verifies via
both createRemoteJWKSet and importJWK paths.
Follow-ups (not in this commit):
- US-6-3: CLI `agentkeys inbox provision|list` wiring
- US-6-4: provisioner-scripts AGENTKEYS_EMAIL_BACKEND selector
- US-6-6: Stage 5b retest against mock-inbox backend
§3 was applying a bucket policy that referenced `role/agentkeys-agent`, but that role is created in §4 — so `put-bucket-policy` failed with `MalformedPolicy: Invalid principal in policy` at §3 execution time because AWS validates principal existence on apply. Split the policy: - §3 now applies `bucket-policy-ses-only.json` with just the AllowSESWriteInbound statement (works with only the SES service principal, no custom-role dependency). - New §4c "Finalize the bucket policy" runs after `agentkeys-agent` is created, overlaying the full two-statement policy. Adds a verification step (`jq '.Statement | length'` should show 2). Inline note in §3 explains the split + offers the alternative ordering (do §4 first, come back). Both orderings terminate in the same state. Caught by user during real-world run; fix is surgical (does not touch §1/§2/§5/§6/§7/§8).
User-requested simplifications to the Stage 6 operator runbook:
1. Swap §3 ↔ §4 ordering: IAM (daemon user + agentkeys-agent role)
now comes first in §3, S3 bucket in §4. This removes the
chicken-and-egg split-policy workaround (no more §4c
"finalize bucket policy" step) because the role exists before
the bucket policy is applied — one put-bucket-policy call with
both statements.
2. Drop the OIDC-federated trust-policy option entirely from the
main runbook. Static IAM user (agentkeys-daemon → AssumeRole →
agentkeys-agent) is the only path. Rationale: OIDC federation
requires (a) oidc.agentkeys.dev hosted publicly with a public-CA
cert and (b) ideally a TEE-derived signer (blocked on
heima-gaps §3). Neither is in the critical path for Stage 6.
3. Remove §5 (IAM OIDC provider registration) from the main doc.
Content preserved in the new demo doc below.
4. Renumber: §6 SES receipt rule → §5, §7 test send → §6, §8
hand-back → §7. Hand-back checklist simplified (drops
TRUST_MODE + OIDC_PROVIDER_ARN; just the static-IAM fields).
5. Cleanup section rewritten: delete daemon user + its access
keys + inline policy; drop the OIDC provider delete; note
that the litentry.org zone itself stays (we only added records,
didn't create a new zone).
6. New docs/stage6-oidc-federation-demo.md — 167 lines — preserves
the OIDC design + end-to-end test for future reference:
- §1 hosting options for the oidc-stub discovery endpoint
(CloudFront+S3, ECS Fargate, ngrok for dev)
- §2 create-open-id-connect-provider
- §3 OIDC-federated role trust policy with agentkeys_user_wallet
PrincipalTag condition
- §4 pointer back to the main runbook for §4 (bucket) + §5-7
- §5 full STS-assume-with-web-identity → temp creds →
per-user-prefix isolation test (including the negative case
where a JWT's wallet tries to read another wallet's prefix
and gets AccessDenied by the bucket policy)
- §6 swap-in path for TEE-derived signer (heima-gaps §3)
- §7 cleanup
The per-user-isolation note in §3b of the main runbook calls out
explicitly that static-IAM means app-side scoping (cloud does NOT
enforce isolation), with a pointer to the OIDC demo for the
cloud-enforced variant. Production workloads should migrate to
OIDC once the hosting + TEE prereqs are in place.
User feedback: the previous standalone docs/stage6-oidc-federation-demo.md
was over-polished for something we cannot yet exercise end-to-end
(needs public OIDC hosting + ideally TEE-derived signer from heima-gaps
§3). Keep the content as a WIP scratchpad we revise as Stage 6 work
progresses, not as a finished manual-test guide.
Changes:
- Delete docs/stage6-oidc-federation-demo.md (167-line polished form).
- Add docs/stage6-wip.md (lighter, clearly WIP-tagged) containing:
- §1 TODO placeholder for the static-IAM Stage 5b-on-SES retest
once AGENTKEYS_EMAIL_BACKEND=ses-s3 is wired and tested live
- §2 TODO placeholder for harness/stage-6-done.sh
- §3 OIDC federation demo — the essential bits preserved (register
OIDC provider, swap to federated trust policy, switch bucket
policy to PrincipalTag scope, run the JWT → STS → own-prefix-ok /
other-prefix-AccessDenied proof). Trimmed of the production
polish. Explicitly notes what's missing (hosting + TEE signer)
and where to swap keys.ts when the TEE side lands
- "Things that moved around" + "TODO pickups" sections so future
sessions can reconstruct the trail
- Update docs/stage6-aws-setup.md cross-references (3 places): links
now point to stage6-wip.md §3 instead of the deleted doc.
The AWS setup runbook (stage6-aws-setup.md) name stays put per user
direction — Stage 5 consumes this infra, and renaming would churn
cross-refs without payoff.
cat > foo.json <<EOF blocks in docs/stage6-aws-setup.md leave ephemeral policy / trust / DNS files in the repo root. They're not meant to be committed — the runbook regenerates them each time. Adding to .gitignore prevents accidental inclusion in future commits.
Architectural re-scoping. The OIDC-federation test is canonically
Stage 7 (Generalized OIDC Provider) per
docs/spec/plans/development-stages.md, not Stage 6 (Federated Own
Email). Stage 6 uses static-IAM-user trust; Stage 7 swaps it for
TEE-signed-JWT-via-PrincipalTag.
Changes:
- git mv docs/stage6-wip.md → docs/stage7-wip.md
- Rewrite the doc: strip the Stage 6 TODO placeholders that
belonged to the prior location. Keep only the OIDC-federation
test content:
* Prereqs: AWS + public OIDC hosting + TEE-derived signer
(each gated on its own dependency)
* Register OIDC provider in IAM
* Update agentkeys-agent trust policy to federated variant
* Upgrade bucket policy to PrincipalTag-scoped
* End-to-end: JWT → STS → temp creds → own-prefix-ok /
other-prefix-AccessDenied proof
* Swap-in note for the TEE-derived signer once heima-gaps §3 closes
- Update 3 cross-refs in docs/stage6-aws-setup.md (audience
paragraph, §3 intro, §3b per-user-isolation note) to point at
stage7-wip.md; reword the audience line to explicitly call it
"Stage 7 work" rather than "future work" for precision.
Per user direction, the AWS-setup runbook file name stays as
docs/stage6-aws-setup.md — it's the Stage 6 infra that the Stage 5
demo consumes, and renaming would churn cross-refs without payoff.
User hit `MalformedPolicyDocument: The policy failed legacy parsing`
on `aws iam put-user-policy`. Root cause: `$ACCOUNT_ID` was unset
in the shell at heredoc-write time (new terminal tab wiped the §0
env setup), so the ARN landed as `arn:aws:iam:::role/agentkeys-agent`
— three colons in a row, empty account segment, AWS rejects with
the unhelpful legacy-parsing error.
Added a sanity-check block at the top of §3a using `${VAR:?msg}`
so the next reader fails loud with a clear "re-run §0" message
instead of the AWS opaque parsing error. Also noted the
`grep Resource *.json` self-check after any heredoc write.
Real-run failure exposed by `cat daemon-user-inline.json | jq .`:
"Resource": "arn:aws:iam::429071895007ole/agentkeys-agent"
^^^ zsh ate the :r from :role
Root cause: zsh parses `\$ACCOUNT_ID:role/...` inside double-quoted
strings as `\${ACCOUNT_ID:r}ole/...` — the :r modifier strips a
filename extension. 429071895007 has no extension, so the modifier
returns the value unchanged, then consumes the r. Result: silently
wrong ARN, rejected by IAM with the cryptic "failed legacy parsing."
Bash does not do this. xxd confirmed the file bytes were clean, so
the bug is at heredoc-expansion time, not file-content time.
Fix: replace every bare \$VAR:<modifier-letter> with \${VAR}:
in both runbook files. Seven instances total across:
- docs/stage6-aws-setup.md lines 178, 191, 224, 273
- docs/stage7-wip.md lines 30, 66, 98
The :r, :h, :t, :e, :l, :u, :a, :A, :p, :q, :x, :s/s/S/P zsh
modifiers all silently corrupt ARNs when a literal colon follows
the VAR inside double quotes. Brace wrapping disambiguates in both
zsh and bash.
Cross-check (grep): no remaining bare \$(ACCOUNT_ID|REGION|DOMAIN|
BUCKET|PARENT_ZONE_ID|OIDC_ISSUER|OIDC_PROVIDER_ARN|ROLE_ARN|WALLET):
pattern in either file.
Zsh modifier bug made `cat > file.json <<EOF ... \$VAR:role ... EOF`
silently corrupt ARNs. Brace-wrapping (${VAR}:) defends against :r
but not against the whole class (stale files on disk, env-lost-between-
shells, invisible Unicode). Switch every JSON-building block in both
Stage 6 docs to `jq -n --arg` piped into command substitution.
Blocks converted (stage6-aws-setup.md):
- §2 DNS change batch (dns-change.json → jq | aws route53 ...)
- §3a daemon user assume-role policy (already inline; rewritten as
jq for consistency with the rest of the doc)
- §3b role-trust.json + role-inline.json
- §4 bucket-policy.json
- §5 SES receipt-rule.json
Plus (stage7-wip.md):
- §2 role-trust-oidc.json
Why jq:
- --arg interpolation happens outside shell parameter expansion, so
zsh modifier shortcuts (:r, :h, :t, :e, :l, :u, ...) can never
corrupt values
- jq validates JSON by construction — no invisible Unicode, no
malformed braces
- command substitution feeds the result straight into the AWS CLI
arg; no file lands on disk, so re-runs can't pick up stale content
Related changes:
- §3a: replaced the heredoc-warning preamble with a shorter env-var
sanity check. Removed the "Why inline instead of file://" note;
the whole doc now has zero files so that explanation is moot.
- §4: dropped the "What's different from the OIDC path" footnote
reference to the old file-based pattern.
- .gitignore: dropped the 10 entries for runbook-generated JSON
files (no longer produced). Keeps AWSCLIV2.pkg (user's CLI
installer artifact, still relevant).
Lint confirmed: zero `cat > ... <<EOF` blocks, zero `file://` refs,
zero bare `\$VAR:<modifier-letter>` patterns remain in either doc.
Also committed to ~/.claude/CLAUDE.md (user-global): a new "Docs &
runbooks — JSON generation pattern" section encoding the rule so
future sessions default to jq from the start.
Real-run hit: user copy-pasted the literal `<most-recent-msg-id>` placeholder into `aws s3 cp`, got `HeadObject 404`. The doc was asking the operator to hand-edit a placeholder — exactly the kind of copy-paste fragility we're trying to remove. Rewrite §6: - Add a sanity-check about macOS `/usr/bin/mail` (no MTA configured by default → silent local-queue dropouts are the #1 "nothing landed" cause for mac operators). Direct them to send from a real outside mailbox instead. - Replace the hand-substituted `<most-recent-msg-id>` with a self-substituting LATEST= query: aws s3api list-objects-v2 ... --query 'sort_by(Contents,&LastModified)[-1].Key' If no objects landed, the query returns the string "None" and we short-circuit with a clear "see troubleshooting below" message instead of dumping a 404. - New "Troubleshooting — nothing landed in S3" sub-block with four diagnostic checks (receipt rule active? MX resolves? SES identity healthy? sender bounced?) so the operator can self-serve before escalating. The only remaining `<msg_id>` ref in the doc (line 299) is in a descriptive sentence about the URL shape, not a copy-paste slot.
User hit two confusion points in §6: 1. Saw `inbound/AMAZON_SES_SETUP_NOTIFICATION` and didn't know it was AWS's own write-access marker (auto-created on receipt-rule activation), not their test mail. 2. Pasted the literal `<most-recent-msg-id>` placeholder into `aws s3 cp` and got a 404. (The runbook's prior version already parameterized this as a `LATEST` query, but didn't filter out the SES marker, so a fresh-bucket run would still pick the wrong key.) Fixes: - Top-of-§6 banner explains AMAZON_SES_SETUP_NOTIFICATION is normal and what it confirms (SES → S3 plumbing). - The LATEST query now uses a JMESPath filter to exclude the marker: `[?Key!=`inbound/AMAZON_SES_SETUP_NOTIFICATION`] | [-1].Key`. So if no real mail has arrived, `LATEST` is `None` and we print a "no inbound mail yet" hint instead of accidentally dumping the marker file. - Added an "Alternative sender — SES self-loop" block: `aws ses send-email` from one address on the verified domain to another. Avoids depending on the operator switching to Gmail mid-runbook. Calls out the SES-sandbox caveat (verify recipient or request production access) since most fresh accounts hit it. The macOS `mail`-doesn't-deliver caveat from the prior version is preserved verbatim — it's a frequent first-run pitfall.
Net-new §7 between Test (§6) and Hand-back (§8 — renumbered from §7). Three subsections covering the operational gaps the wildcard receipt rule leaves open: - §7.1 S3 lifecycle policy — auto-expire `inbound/*` after 30 days. Single jq-built `put-bucket-lifecycle-configuration` call. Throwaway inboxes accumulate verification mails + any spam that slips through; this caps storage growth without per-message logic. Tunable via the Days field. - §7.2 Spam handling — handle at READ time in the daemon, not at SES write time via Lambda. Daemon parses `X-SES-Spam-Verdict` from the .eml and drops on FAIL. Keeps the receipt rule trivial, avoids per-message Lambda invocation cost, and pushes the policy decision to where it belongs (the bot expects ONE specific verification mail; anything else is junk regardless of SES's verdict). Pseudo-code included. Adding a write-time Lambda is flagged as scale-only. - §7.3 SES sandbox vs production — sandbox restricts OUTBOUND only, not inbound. The Stage 6 demo's inbound-only flow works in sandbox. Production access is only needed when the agent itself sends mail to arbitrary user addresses. AWS Console request, ~24h review. Plus an explicit "What we're NOT mitigating" subsection for items deferred to Stage 6 post-MVP: address enumeration defense, per-recipient inbound rate limit, sender allow/deny lists. Each lists the Lambda pattern that would close it. §8 Hand-back content unchanged; just renumbered from §7.
Two parallel inserts answering "how does the AWS stack actually map
to AgentKeys users at runtime?" — the question that surfaced after
the user worked through stage6-aws-setup.md and saw the singleton
agentkeys-daemon + agentkeys-agent design.
wiki/email-system.md (published — paranoid about secrets):
New section "Architecture topology — singleton scaling model"
between §"SES architecture, one page" and §"How we differ from
AgentMail". Mermaid + ASCII diagram of the resource graph. Tables
for singleton-vs-per-user, AWS-quota math, trust chain, and
Stage 6 vs Stage 7 isolation comparison. All ARNs use
<ACCOUNT_ID> / <DOMAIN> placeholders — zero leak of the operator's
account ID, hosted zone ID, or bucket name.
docs/spec/ses-email-architecture.md (in-repo spec — engineer-deep):
New §6.5 "Architecture topology — singleton resources, logical
per-user separation" between §6 (Receive pipeline) and §7 (Send
pipeline). Same Mermaid + ASCII diagram, plus deeper IAM-trust-
chain detail (full inline-policy structure for both user + role,
trust-policy reciprocal). Same placeholder discipline.
Both inserts cover:
- Resource graph (mermaid for GitHub render, ASCII for terminals)
- Singleton vs per-user table
- AWS-quota analysis (why per-user IAM caps at 1k users)
- Storage math at 10k users / steady state
- Trust chain narrative (operator key → IAM user → AssumeRole →
role → 1h temp creds → API)
- Stage 6 vs Stage 7 per-user-isolation comparison
Lint pass: grep -E '429071895007|Z09723983CFJOHAE3VC65|agentkeys-mail-[0-9]+'
returns zero matches in both files. The operator's account ID, hosted
zone ID, and real bucket name remain confined to docs/stage6-aws-setup.md
(operator runbook, in-repo only) where the literal values are needed
for copy-paste execution.
Append §5 "Backend selector" to manual-test-stage5.md. Companion to the US-6-4 backend-selector work in provisioner-scripts (commits e958869, 24744dc, d1ca652) that added gmail / mock-inbox / ses-s3 dispatch in provisioner-scripts/src/lib/email.ts. Lists the three accepted values, the env vars each requires, and a quick smoke test for the mock-inbox dispatch path. Stage 5 demo readers moving toward Stage 6 see the env-var surface in one table without having to grep the TS source. Originally added by the executor agent during the Stage 6 ralph session; got bumped past the prior commit boundaries. Landing now as a standalone doc commit.
New docs/manual-test-stage6.md — companion to manual-test-stage5.md,
with the exact commands to run now that the user has the AWS stack
standing (per stage6-aws-setup.md).
Flow:
1. Build binaries + install deps (one-time)
2. Shell env setup — including the critical `sts:AssumeRole` step
that swaps static daemon-user keys for 1h temp creds with real
S3+SES permissions. Without this step the ses-s3 backend gets
AccessDenied on every ListObjects.
3. Start mock server (Terminal A)
4. Launch real Chrome with --remote-debugging-port=9222 for the
CDP scraper to connect to (Terminal B)
5. agentkeys init --mock-token stage6-demo
6. Run openrouter-cdp.ts directly, capture sk-or-v1-* key on stdout
7. agentkeys store openrouter <key> + curl /api/v1/models verify
Calls out the known demo-time limitations explicitly:
- agentkeys provision openrouter still invokes openrouter.ts
(Turnstile-blocked headless path), not openrouter-cdp.ts. User
runs the CDP scraper directly and pipes the key into store.
Fix-later is a --cdp flag on provision.
- AssumeRole is manual. Fix-later is in-daemon AssumeRole +
auto-refresh.
- AMAZON_SES_SETUP_NOTIFICATION is polled every cycle but filtered
out by from/subject regex — benign.
Verified before ship:
- All 6 cross-referenced files exist (stage6-aws-setup.md, wiki/
email-system.md, spec/ses-email-architecture.md, stage7-wip.md,
ses-s3.ts, openrouter-cdp.ts)
- All 8 env var names in the doc match what the code reads
(grep -rnE 'AGENTKEYS_*' provisioner-scripts/src/)
- Both release binaries (agentkeys, agentkeys-mock-server) exist
- `agentkeys init --mock-token <tok>` syntax matches CLI --help
User has two distinct sets of AWS keys in env: AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY for agentKeys-admin (the IAM user that built the infra), and DAEMON_ACCESS_KEY_ID + DAEMON_ACCESS_KEY_SECRET for agentkeys-daemon (the IAM user the daemon impersonates). Updates: - §2 split into 2a (always-on Stage 6 vars), 2b (preflight: confirm admin creds work via aws sts get-caller-identity), 2c (switch to daemon creds + AssumeRole → 1h temp creds, with a third get-caller-identity to confirm the assumed-role ARN). - §2c saves admin creds to ADMIN_AWS_ACCESS_KEY_ID and ADMIN_AWS_SECRET_ACCESS_KEY before swapping in daemon creds. - §Teardown now has a "Restore admin creds" block that reads from the same ADMIN_AWS_* vars, with a fourth get-caller-identity to confirm the swap-back lands. - Inline note that AWS-standard secret-half env var name is AWS_SECRET_ACCESS_KEY, not AWS_ACCESS_KEY_SECRET (typo trap; the user mentioned the wrong name in conversation, so the doc preempts the silent SDK fallback that follows from the typo). - Cleanup also clarifies AWS-infra-teardown should NOT run between demo iterations — re-running §2c gives fresh temp creds. Verified: cross-refs all exist; env var names consistent across all four occurrences (L23, L68-69, L72-73, L206-207); section structure flows 1 → 2[abc] → 3 → 4 → 5 → 6 → 7 → 8 → Limitations → Teardown → Cross-references.
User reset their env scheme so AWS_* is never set persistently: agentKeys-admin → ADMIN_AWS_ACCESS_KEY_ID + ADMIN_AWS_ACCESS_KEY_SECRET agentkeys-daemon → DAEMON_ACCESS_KEY_ID + DAEMON_SECRET_ACCESS_KEY Under this scheme, AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN only ever hold the 1h assumed-role temp creds. The prior §2 had three subsections (2a setup vars, 2b translate non-standard secret name + confirm daemon, 2c AssumeRole with save/restore) which was necessary when daemon creds were in AWS_* but is pure noise now. Changes: - Collapse §2a/§2b/§2c into ONE copy-pasteable block. Uses env-prefix (`AWS_ACCESS_KEY_ID=$DAEMON_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$DAEMON_SECRET_ACCESS_KEY aws sts assume-role ...`) so daemon creds are scoped to that one subprocess only. Temp creds then export into AWS_*. Single `get-caller-identity` + S3 list sanity check at end. - Drop the `AWS_ACCESS_KEY_SECRET` vs `AWS_SECRET_ACCESS_KEY` gotcha note — no longer relevant since the user's daemon creds live in DAEMON_* names that they control. - Drop DAEMON_SAVED_* stash — env-prefix pattern needs no save/restore. - Simplify §Teardown: no admin-restore into AWS_* (AWS_* never held admin creds); just `unset` the temp creds and use env-prefix for any admin commands you want to run after. - Keep the one-line naming-drift note so the asymmetric `_ACCESS_KEY_SECRET` vs `_SECRET_ACCESS_KEY` doesn't bite later. Net change: 36 insertions, 64 deletions. Doc 201 lines (was ~229). Section structure flat: 1 → 2 → 3 → ... → 8 → Teardown → Cross-refs.
…ice signup automation Adds the agentkeys-workflow-collection skill that records signup + API-key-mint flows end-to-end against real Chrome over CDP, plus chrome-devtools-mcp wired in via .mcp.json so the diagnose loop runs inside Claude's tool surface instead of throwaway probe-*.mjs scripts. Working end-to-end (validated this session): OpenRouter (sk-or-v1-*), OpenAI (sk-proj-*). Brave Search clears every anti-bot layer (PoW, email verify, login, 2FA OTP) and stops only at the paid-plan paywall. Recorder additions - workflow-recorder/ — full skill: flows.ts, email-analyzer.ts, turnstile-handler.ts, hcaptcha-handler.ts, artifacts.ts emitter, record-service.ts CLI, credential-resolver.ts. - Multi-step signup support (email -> password page handoff for OpenAI Auth0). - Post-verify profile fill (OpenAI /about-you with name+age+blur). - OAuth callback settle wait before keys-page navigation. - DOM-direct chained-modal dismissal (OpenRouter survey + welcome banner). - OTP fallback extraction when analyzer misclassifies as magic-link (Brave 2FA email contains a verification URL that trips magic-link regex). - React/Svelte-aware OTP fill via pressSequentially + dispatched events. - Turnstile detect uses count() fallback for shadow-DOM rendering. - hCaptcha handler with CapSolver provider (account does not currently support hCaptcha service; handler falls back cleanly to human-in-loop). - emitDraftScraper writes self-contained draft scrapers with no TODOs. Infra - scripts/reset-chrome-for-recording.sh — single Chrome launcher (parity with old manual contracts plus idempotent kill+wipe+CDP-wait). - scripts/stage6-demo-env.sh, stage6-demo-run.sh, stage6-inspect-email.sh — Stage 6 operator helpers. - .mcp.json — chrome-devtools-mcp attached to existing Chrome on :9222. - agentkeys-secrets.env.example — env template (DAEMON keys, signup creds, optional CAPSOLVER_API_KEY). - archived-probes/ — historical one-off probes preserved for reference; superseded by mcp__chrome-devtools__* tools in the diagnose loop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extracts the two proven signup flows from the recorder into self-contained
per-service scrapers. Users' agents call node openrouter-cdp.ts / openai-cdp.ts
with env vars, get back a minted sk-* key — zero human intervention on
happy path.
- src/scrapers/openrouter-cdp.ts rewritten with this session's fixes
(chained-modal dismiss, Clerk __clerk_status=verified handling,
force-click over mid-fade-in overlays). 30s end-to-end.
- src/scrapers/openai-cdp.ts new. Multi-step email→password→OTP→
/about-you profile→OAuth callback→instant-mint. 73s end-to-end.
Uses fetchAndAnalyzeSesEmail for label-aware OTP extraction so CSS
hex colors don't false-positive as the OTP.
- src/lib/playwright-patterns.ts new: generic primitives shared between
recorder and scrapers (humanType, clickFirstVisible, clickOuterCreate
with onBeforeIteration callback, dismissCookieBanner,
probeAndDismissDialog, isDialogOpenByText, jitterDelay).
- src/lib/captcha/{turnstile,hcaptcha}.ts moved from workflow-recorder/.
- src/lib/email-analyzer.ts moved from workflow-recorder/, added
fetchAndAnalyzeSesEmail high-level helper.
- src/workflow-recorder/flows.ts refactored to import generic helpers
from lib/. Service-specific onboarding dismiss stays inline (review
decision: lib stays generic, service-specific knowledge lives in callers).
- scripts/weekly-live-test.sh: runs both scrapers live against real
providers, reports pass/fail. NOT in CI — for the project owner / human
contributor to run periodically to catch provider-side flow drift.
- archived-probes/: 5 diag-*.mjs moved out of provisioner-scripts/ root.
Output contract (matches Rust spawn_and_collect IPC): newline-delimited
JSON events on stdout. Progress {"type":"progress","step":"<label>"},
terminal {"type":"success","api_key":"sk-..."} on success or
{"type":"error","code":"...","details":"..."} on failure. Shell users
pipe through jq -r 'select(.type=="success") | .api_key'.
Verified this session:
- tsc --noEmit clean
- recorder parity: sk-or-v1****38db
- openrouter-cdp.ts: sk-or-v1-c7d88ac5...647d47 in 30s
- openai-cdp.ts: sk-proj-3YBTQnH-8qPx...E3IA in 73s
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 23, 2026
Closed
hanwencheng
pushed a commit
that referenced
this pull request
Apr 23, 2026
Replaces the 1623-line stage-by-stage history with a Shipped / Active /
Planned summary; adds a single CDP-only dev-setup + demo guide;
archives the drifted manual-test-*.md per-issue files and the
Gmail-backed demo variants.
- `docs/spec/plans/development-stages.md` — rewritten (~180 lines).
Shipped: Stages 0-5a + 6 (interim) each as one-line summaries.
Active: Stage 5b telemetry + LLM-fallback plan, Stage 6 finalization.
Planned: Stage 7 OIDC, Stage 8 Priority A, npm packaging.
- `docs/dev-setup.md` — CDP-only onboarding guide. Prereqs → build →
Stage 6 env → OpenRouter demo → OpenAI demo → verify → troubleshoot.
Explicitly dismisses the Gmail demo variant (archived).
- `docs/archived/` — 13 files moved (development-stages v1, four
manual-test-stage{4,5,6} / stage5-workspace-email-setup, and the
eight per-issue manual-test-issue-1X.md + report). README.md spells
out the supersession mapping + archive policy.
- `.gitignore` — block Stage 6 one-shot JSON artifacts from creeping
back (`bucket-policy*.json`, `daemon-user-inline.json`,
`dns-change.json`, `probe*.mjs`) per CLAUDE.md's jq-only rule; ignore
the double-nested `provisioner-scripts/provisioner-scripts/` tsx
cwd artifact.
- `CLAUDE.md` — keep the "do not read folder docs/archived" hint.
Follow-up to #52. No runtime code changes; docs + gitignore only.
hanwencheng
added a commit
that referenced
this pull request
Apr 23, 2026
#53) Replaces the 1623-line stage-by-stage history with a Shipped / Active / Planned summary; adds a single CDP-only dev-setup + demo guide; archives the drifted manual-test-*.md per-issue files and the Gmail-backed demo variants. - `docs/spec/plans/development-stages.md` — rewritten (~180 lines). Shipped: Stages 0-5a + 6 (interim) each as one-line summaries. Active: Stage 5b telemetry + LLM-fallback plan, Stage 6 finalization. Planned: Stage 7 OIDC, Stage 8 Priority A, npm packaging. - `docs/dev-setup.md` — CDP-only onboarding guide. Prereqs → build → Stage 6 env → OpenRouter demo → OpenAI demo → verify → troubleshoot. Explicitly dismisses the Gmail demo variant (archived). - `docs/archived/` — 13 files moved (development-stages v1, four manual-test-stage{4,5,6} / stage5-workspace-email-setup, and the eight per-issue manual-test-issue-1X.md + report). README.md spells out the supersession mapping + archive policy. - `.gitignore` — block Stage 6 one-shot JSON artifacts from creeping back (`bucket-policy*.json`, `daemon-user-inline.json`, `dns-change.json`, `probe*.mjs`) per CLAUDE.md's jq-only rule; ignore the double-nested `provisioner-scripts/provisioner-scripts/` tsx cwd artifact. - `CLAUDE.md` — keep the "do not read folder docs/archived" hint. Follow-up to #52. No runtime code changes; docs + gitignore only. Co-authored-by: wildmeta-agent <wildmeta-agent@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes Stage 5 and Stage 6 for the v0 critical path. Ships three deliverables:
bots.litentry.org— fully owned throwaway-inbox infrastructure, replaces the Gmail+plus-addressing demo path. Docs at docs/stage6-aws-setup.md, spec at docs/spec/ses-email-architecture.md.agentkeys-workflow-collectionskill (~/.claude/skills/) + recorder underprovisioner-scripts/src/workflow-recorder/— dev-time iteration tool that drives real Chrome via CDP, captures per-step artifacts, emits draft-scrapers. Chrome-DevTools MCP integration via.mcp.jsonreplaces throwawayprobe-*.mjsscripts with live in-loop diagnostics.src/scrapers/openrouter-cdp.ts(sk-or-v1-*, ~30s) andsrc/scrapers/openai-cdp.ts(sk-proj-*, ~73s). Output newline-delimited JSON events matching the Rustspawn_and_collectIPC contract.Shared generic helpers lifted into
src/lib/playwright-patterns.ts,src/lib/captcha/, andsrc/lib/email-analyzer.ts. Service-specific bits (OpenRouter onboarding-modal dismiss, OpenAI/about-youprofile fill, label-aware OTP extraction) stay in the scraper files per review decision.Supersedes #47 (Stage 5a harness refactor — content already on this branch via #48) and #49 (Stage 5 demo runbook — subsumed by this branch's Stage 5 + Stage 6 docs).
Test plan
npx tsc --noEmit— clean across workflow-recorder + scrapers + libcargo test -p agentkeys-mock-server— mock inbox handler tests passcargo test --workspace— full Rust suiteagentkeys-workflow-collection→ OpenRouter signup →state: completed,sk-or-v1****38dbin 36snode src/scrapers/openrouter-cdp.ts→sk-or-v1-c7d88ac5...647d47in 30s (JSON events, exit 0)node src/scrapers/openai-cdp.ts→sk-proj-3YBTQnH-...E3IAin 73s (JSON events, exit 0)bash -n provisioner-scripts/scripts/weekly-live-test.sh→ exit 0Follow-ups (tracked separately)
Manifestinterface, rewriteemitDraftScraperto compose from lib calls, add new/agentkeys-ship-scraperskillprovisioner-service=openai|openrouterat the new scrapers)🤖 Generated with Claude Code