feat(web-sources): crawl websites into an assistant's knowledge base

[philmerrell]

Summary

Adds an "Add web content" flow in the assistant editor that lets users attach a single web page or a bounded crawl of a site to the assistant's knowledge base. Discovered pages flow through the existing S3 → ingestion Lambda → chunking/embedding pipeline — no new infra.

• New backend package `apis/app_api/web_sources/` (BFS crawler, repository, routes, URL/SSRF helpers).
• New SPA `WebSourceDialogComponent` with single-page and crawl modes (depth, max pages, concurrency, delay sliders); editor wires it as a sibling to the connector buttons.
• Two new pinned deps: `beautifulsoup4==4.13.5`, `trafilatura==2.0.0`.
• Drive-by UX fix on the existing docs list: delete is now optimistic (row removes instantly, rolls back on failure) and works on stuck-uploading docs.

Behind the scenes

• Crawler respects robots.txt, stays same-domain, has per-host jitter + bounded concurrency, SSRF-guards every URL, 5 MB per-page cap, 15-minute crawl budget, always finalizes in a `finally`.
• `CrawlJob` rows live in the assistants table at `SK=CRAWL#{crawl_id}`, get a 30-day TTL on terminal status, and cascade-delete when the last web doc for that root URL is removed.
• `list_active_crawls` self-heals — a `running` row older than 20 minutes (the crawler's budget + 5 min buffer) is auto-finalized, so a crashed process can't leave the SPA in perma-poll.
• Crawler workers and the route-level background task both hold strong refs, sidestepping Python's weak-task-tracking GC.
• Floats in `CrawlSettings` (min/maxDelay) are coerced to `Decimal` before DynamoDB writes — boto3 rejects bare `float`.

Decisions worth flagging

• HTTP-only for v1. Static HTML → markdown via trafilatura (BS4 fallback). JavaScript-only sites render as empty pages; called out in the modal. Browser rendering via AgentCore browser sessions is a follow-up.
• Submit-and-watch UX rather than discover-then-pick — the modal closes on Start; pages stream into the docs list as they're ingested via an incremental discovery merge (no list-wide refresh).

Test plan

• `cd backend && uv sync --extra agentcore --extra dev && uv run pytest tests/apis/app_api/web_sources/ -v` — 60 tests
• `cd backend && uv run pytest tests/architecture/ tests/routes/test_documents.py` — boundary + adjacent docs routes still green
• `cd frontend/ai.client && npx tsc --noEmit` — clean
• Manual: single-page import of a Wikipedia article → doc reaches `complete` and is citable
• Manual: small-site crawl with depth 2 / max 10 → pages stream into the list, "Crawling…" badge clears on finish
• Manual: 404 URL → root doc transitions to `failed` with a readable error
• Manual: SSRF guard rejects `http://127.0.0.1/...\` with 422 in the modal
• Manual: stale `running` CrawlJob (from a crashed process before this PR) gets reaped on next watcher tick and polling stops
• Manual: delete a complete web doc → row disappears immediately, S3 markdown + vectors cleaned up, orphan CrawlJob row goes too

Out of scope (follow-ups)

• JS-rendering via AgentCore browser (separate plumbing into app-api).
• Cancel-crawl button (crawler would need to poll its own status between fetches).
• Sitemap.xml-driven crawls and scheduled re-crawls.

🤖 Generated with Claude Code