Sell the directory as a corpus, on a four-hour clock

[ralyodio]

/sales and a licensed bulk-download surface for training-data buyers. The open directory is untouched: every endpoint that answered a stranger yesterday still does, and llms.txt now says so explicitly.

What's here

/sales — a public page with live figures read from the directory (no invented numbers, same cached read /advertise and /crawlstats use) and a contact form. No prices, no rate card. Licensing is negotiated per buyer, so a published figure would be either wrong for everybody or a fiction with "contact us" under it. Enquiries land in dataset_enquiries before the notification email is attempted — sendEmail reports failure rather than throwing, and a sales enquiry lost to a Resend outage is the most expensive message this site can drop. Honeypot field plus a 3-per-hour-per-address cap.

Four gzipped-NDJSON streams at /api/dataset/{feeds,items,extracts,authors}, cut on fixed 4-hour UTC boundaries.

The boundaries are the product rather than the packaging. A window is the same set of rows whoever asks and whenever they ask, so a buyer walking them in order provably sees every row exactly once — no gaps from clock skew, no duplicates from a retry, and a broken pull is simply repeated. Only closed windows are served; the one containing the present is still filling, and serving it would promise a completeness it does not have. An unaligned or future boundary is refused rather than rounded, because rounding mislabels the buyer's corpus in a way that only surfaces months later.

Streamed from a keyset cursor rather than built into a file: there is no object storage here, and a periodic dump job would be a large read against a write-saturated database producing an artifact most windows nobody collects. Cursors ride (created_at, rowid) / (status, fetched_at, rowid) on indexes that already exist — a cursor on (created_at, id) would have needed a new index built over 4.7M rows at poller boot.

/api/dataset — an open manifest describing every stream, the cadence and the current boundary. Deliberately no row counts: it is unauthenticated, and a count(*) over feed_items would hand anybody a way to make the site slow.

Access is a hand-written row in dataset_grants after a conversation. There is no endpoint anywhere that mints one, and that is the intended amount of friction. A session or an API key both work; the licence hangs off the account, so a buyer can rotate keys without telling us. Refusals are distinguishable — 401 (sign in) / 401 (bad credential) / 402 (no licence, here is /sales) — because they need three different actions.

Two things worth reviewing closely

Post records ≠ article text, and the page says so unprompted. content_html stopped being written in 0031_item_body_on_demand.sql, so the 4.7M figure is metadata and the prose lives in item_extracts (~275k articles, ~7.8k chars each, growing with reader attention rather than with the crawl). A buyer who learns this after signing is a refund; the sales page and the manifest both lead with it.

A publisher can now opt out of the corpus while staying in the directory. feeds.dataset_opt_out, honoured by all four streams — including extracts, which reaches the flag two joins away, and there is a test for exactly that. Today the only way to express "list my blog, but don't sell my writing to a model" is to ask for removal, and the directory loses a good feed over a question it never asked.

authors.email is never emitted. The column holds addresses people published as their own contact; a corpus of contactable humans is a different product from a corpus of what they wrote.

Verification

• 19 new packages/db tests, 11 new apps/web tests. Full suite 1,325 passing on Node 22 (matching CI), pnpm build clean.
• End-to-end smoke test against a local SQLite database: every refusal path (401/401/402/404, unaligned, still-filling), the per-window limit (2 allowed → 3rd is 429), a different window keeping its own allowance, the full-dump day limit, real gzip bytes on the wire, the audit log recording start-before-finish, and the contact form in both form-encoded and JSON modes.
• The smoke test caught a real bug the unit tests missed: the authors stream queried author_links.kind, which does not exist (it is network). An empty fixture table cannot fail a query. Fixed, and the test that would have caught it is now in the suite — the links now also carry handle, source and the verified rel="me" flag.

Deploying

The migration applies itself at poller boot. alter table feeds add column with a constant default is a schema-only change in modern SQLite, so it does not rewrite 535k rows. Nothing else is needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Pij2tFcRgqceMpheotFSoX