Site Recipes

Site Recipes

sites/ in the repo is navette's navigation memory: one folder per site navette already knows, holding a machine-readable recipe and the content acquired through it.

sites/
  dev.to/recipe.md  pages/…
  daily.dev/recipe.md  pages/…

Why recipes exist — measured

Task: list the 5 latest dev.to notifications with dedup ID, type, actor, deep link, relative time — logged-in session, identical human pacing (n=1, hand-driven, 2026-10-13):

Metric	WITHOUT recipe (cold exploration)	WITH recipe.md
HTTP calls to the daemon	6	4
Wall clock (throttle included)	23 s	5 s
Bytes daemon → agent	109,296	1,326 (−98.8%)
~Tokens ingested by the agent	~27,000	~330
Stable dedup IDs	no — markdown doesn't carry data-notification-id	yes, from the first call
Output	markdown to re-parse + improvised DOM probing	verified JSON, ready for dedup

The call-count gap is modest; the real win is ingested load (−98.8% bytes, ~−99% tokens) and output reliability. A smart cold agent can shave one call and still land 60× heavier, with no usable IDs. The bench doubles as the repair loop: the recipe's first pass shipped without the JSON.stringify wrapper → the macOS WebKit bridge returns plist, not JSON — fixed in the recipe before commit. Initial recipe cost: one live recon (~15 calls, once per site).

recipe.md — the contract

The reader is an agent (LLM) driving navette over HTTP — not a parser. So: zero prose, stable keys, every line = one call or one selector. Extraction JS is verified live before being committed — if it hasn't run, it doesn't belong in the recipe.

Required head keys:

Key	Meaning
domain, base	the site and its base URL
auth	how a session becomes logged-in (keychain store name + /sessions/load; login wall → /sessions/show)
render	page class (static / semi-JS / SPA) and the sync rule (always /wait the ready selector after navigate)
throttle	non-negotiable — human speed (4–9 s between navigations), slow acquisition
checked	date + how selectors were verified (scratch daemon, keychain store, cookie count)
acquire	READ-ONLY — no action clicks (like, follow, publish, upvote, bookmark); /click reserved for navigation flows

Invariants (every site)

• The recipe's throttle is non-negotiable — human speed, slow acquisition.
• Acquisition is read-only: no action clicks, ever.
• Never a captcha: POST /sessions/show and the human takes over.
• A selector breaks → re-scout one page (outerHTML probe), patch the recipe, commit.
• The hard close-of-pass rule: a pass is not finished until the recipe is updated and committed. What actually ran = live-verified = goes in; assumptions that never executed stay out. An update that isn't committed is a lost pass.

Acquired pages

1 page = 1 file pages/<type>/<YYYY-MM-DD>-<slug>.md:

---
url: https://dev.to/...
captured: 2026-10-13
hash: <content sha>
type: notification | article | comment | feed-index
---
<parsed content>

Dedup without a database: grep -rl "<url>" pages/ — present and identical hash → only touch captured.

dev.to recipe (summary)

• Auth: keychain store devto → /sessions/load {"session":S,"store":"devto"}; login wall → /sessions/show.
• Render: semi-JS — always /wait the flow's ready selector (~1–2 s).
• Throttle: 4–9 s between navigations; /evaluate//read on the current page: no throttle.
• Flows: feed (/), notifications (/notifications, auth — .spec-notification[data-notification-id], ~38 per first page, no HTTP pagination), article (#article-body, comments under .comment[data-comment-id] with data-path anchors), search (/search?q=, JS-rendered, 0 is a valid answer), profile (/<username>).
• Key selectors: feed cards article.crayons-story[data-feed-content-id]; notif deep link header + p.lh-tight a → /<user>/comment/<ID>; tag windows /t/<tag>?top=week|month|year.
• Extraction: one FEED_JS / NOTIF_JS / ARTICLE_JS blob per flow, all wrapped in JSON.stringify (MANDATORY — plist bridge), saved as typed pages with stable dedup IDs.

daily.dev recipe (summary)

• Auth: dedicated logged-in session on the shared daemon — use READ-ONLY, never /sessions/close it (n8n workflows depend on it); keystore builds: GET /states then /sessions/load {"store":<name>}.
• Render: SPA (Next.js) — /wait the ready selector (~2–4 s hydration). This daemon build's /wait accepts selector only ({"ms":N} returns {"ok":false}).
• Routes: feed /, post /posts/<slug>-<id> (short form /posts/<id> resolves), search /search?q=, profile /<username>/posts, bookmarks /bookmarks (auth), notifications /notifications (auth).
• Key selectors: feed cards article.Card_post__S2J45 (plain article matches landing widgets — too early); post ready selector is #comment-post-btn (h1 is MISSING on aggregator-summary posts — never wait on h1); comment dedup key from button[id$="-upvote-btn"] (comment- prefix / -upvote-btn suffix stripped); comments mount lazily below the fold — /scroll {"y":900} then ~3 s; ad promos carry span.commentAuthor — filter on the anchor a.commentAuthor.
• Gotchas: payload discipline (build JSON in a file, validate with python3 -m json.tool — a malformed body makes /evaluate return {"ok":true,"result":""} silently); the bridge drops occasional evaluates during hydration — retry once; prefer single quotes in JS strings so only JSON-level " needs escaping; dedup crossposts on orig (the api.daily.dev/r/ redirect), not on href.

Adding a site

1. Live recon through navette (/evaluate probes, real selectors, never guessed).
2. Write sites/<domain>/recipe.md with the head keys + flows.
3. Commit — the folder is versioned; a PR is enough to grow the memory.

The bench harness for the recipe-vs-cold comparison lives in bench/recipe/ (bench-notifications.sh replays the "with" side in one shot). Numbers: Benchmarks.