Count what is actually asking, before choosing what to charge it

[ralyodio]

Why

We have ~1M requests a month and 14 accounts, 1 of them active besides Anthony. That gap is the thing to act on, but "1M pageviews" is not one fact — it is three populations summed together:

• people reading, who must never meet a rate limit;
• agents calling the API and /mcp on purpose, which is the product working and the thing a sponsorship would charge for;
• scrapers walking all 37 HTML pages to rebuild what /api/feeds hands over in one request — pure cost, no relationship.

The tiering we want (free anonymous allowance, ~10x for signed-in readers, sponsorship above that) is a set of numbers somebody has to pick, and picking them blind fails both ways: under what a reader does in an hour and the directory breaks for people; over what a scraper does and it costs exactly what no limit costs.

So this PR measures. It changes no limit and refuses no request.

What is here

• traffic_hourly — one row per hour per (agent, bucket, tier), accumulating upsert. Shaped like the existing crawl_hourly / queue_hourly rollups.
• apps/web/src/lib/traffic.js — the classifier and an in-memory counter. Pure: no Next, no request object, no database, which is what makes it testable.
• apps/web/src/lib/trafficCounter.js — the wiring, called first thing in proxy.js.
• GET /api/traffic?hours= — byAgent, byBucket, byTier, byHour, and readerByAgent broken out on its own.

Design notes

Buffered, not per-request. A write transaction on this database can take 29–300s under crawl load. A row per request would be tens of thousands of transactions an hour competing with the poller for one lock. The counter flushes once a minute — an hour of traffic is 60 batches, and the worst case is losing a minute of counters, which for choosing a rate limit is nothing.

Counting runs before the throttle decides, so a refused request is still counted. A limit that hides the traffic it turns away cannot be tuned against anything.

Browser detection runs last. Nearly every crawler also says Mozilla/5.0 and several name Chrome or Safari deliberately. There is a test asserting six real crawler UAs do not classify as browser — get that order wrong and a million scraper requests read as an audience, which is roughly the mistake this PR exists to stop us making twice.

AI crawlers split by vendor; SEO scrapers kept apart from search. Googlebot sends readers, Google-Extended trains on us, AhrefsBot does neither. "Should this pay" is a question asked one crawler at a time, and a taxonomy that cannot separate them forces one answer for all of them.

Folded into the existing proxy.js rather than added beside it — Next 16 allows one proxy file per app, and that file already runs in front of every request for the crawl throttle. Every path through the counter is wrapped and swallowed: it runs ahead of every response on the site, and an outage caused by bookkeeping is the least defensible kind there is.

Privacy

Records a user-agent family and a route kind. Never a raw UA string, never an address, never who read what. /api/traffic is open and metered by the same guard() as the rest of the API — there is nothing in it that identifies a caller.

What this does not do

• Changes no limit. FREE_PER_WINDOW is still 120/min, signed-in is still unmetered, API keys still 5000/hr.
• A keyed API call currently counts as anon — the key is read in apiguard, after the proxy has run. So anon is an upper bound, and closing that gap is the first job of the tiering work.

Testing

• 18 new tests in apps/web/test/traffic.test.js, built on real UA strings as sent.
• Full workspace suite green: 12 packages, 0 failures (322 in apps/web).
• pnpm build passes with the proxy registered and /api/traffic in the manifest.
• Migration and the accumulating upsert smoke-tested against a local SQLite file: two flushes of the same key sum to 800 rather than replacing.

After merge

The poller applies the migration on boot. Give it a few days, then GET /api/traffic?hours=168 is the input to the tier numbers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SisjDW4F1zwTeySnUM6VbB