Reddit at /r/, X at /x/, and a provider stack behind X

[ralyodio]

Reddit now lives under /r/ and X under /x/, and X is collected through a
replaceable provider stack rather than not existing.

The two halves, and why they are different sizes

Reddit needed a name, not a mechanism. It publishes real RSS, so a subreddit
already resolves down the ordinary path — and lands as an untyped row at a slug
of its own among the blogs. That is how 50,099 of them got here, 41% of the crawl
queue. The migration gives each an identity (r:sub:programming) and /r/programming
answers. Nothing is moved, renamed or deleted: /{slug} keeps working for every
link already pointing at it, and the /r/ address is the canonical one.

X needed the rest of it. X publishes nothing, so a timeline has to be
collected and mirrored:

RSSHub (primary) → Teapot (fallback) → official X API (paid) → the items we hold

Failover is per attempt and the order is fixed; three failures in a row set a
provider aside, one success clears it. Sessions rotate least-recently-used with
per-state cooldowns, and a rate limit, an auth failure, a provider outage and a
missing account are four different things that get four different responses.

The provider appears in no public URL. /x/OpenAI.rss is the same address
whichever of the three filled it, so the collection method can be replaced under
a live subscriber without their reader noticing. There is a test that greps the
rendered document for provider names.

The design decision worth arguing with

The PRD sketches a sources + items pair (§20). That pair already exists here
— it is feeds + feed_items, carrying dedupe, backoff, interval learning,
keyword extraction, FTS, alerts, sitemaps and five syndication formats. So a
social source is an ordinary row in feeds with two extra columns, and crawlFeed
gains a third ingestion path beside resolve and scrape.

The payoff is §30 and AC-8 for free: an X post reaches /topics/*.rss beside a
blog post, and there is no line in the topic path that knows X exists.

source_kind was the obvious column and is deliberately unused — it carries a
CHECK constraint, and SQLite cannot widen one without rebuilding a thirty-column
table with a dozen indexes against 300k rows on a single-writer database.

What is here

• packages/social — canonicalisation for both platforms, X providers, session
  pool, registry, post normalisation. No database client; the stores are injected.
• Migration — social_network/social_ref/social_config, x_provider_state,
  x_sessions, and a two-pass backfill of the existing subreddits. No
  credential column, and no room for one: auth_token/ct0 are a full login
  and stay in the environment, so a leaked dump carries none.
• Crawl — five-minute floor for provider-collected sources rather than the
  directory's hourly one.
• Web — /r, /r/<sub>, /r/u/<user>, /x, /x/<handle>, /replies,
  /media, /x/list/<id>, /x/search, /x/status; each in .rss, .atom,
  .json and .md. A source not in the directory yet gets an offer to add it
  rather than a 404.
• Rivers now answer If-None-Match with a 304 — computed before the ad fetch, so
  an unchanged feed never meters an impression it did not deliver.

Failure behaviour

Nothing in this path can empty a feed. Items are written only on success, so an
upstream outage leaves yesterday's posts exactly where they were and the public
route goes on serving them (§40, AC-5). A rate limit, a provider outage and a
missing X runtime all reschedule without touching a health column — otherwise
markCrawlFailure's ten-strike rule would retire the whole X directory over an
afternoon with no clue why.

Not built

The /admin/x buttons of §34 — disable a provider, clear a cooldown, force a
refresh. This codebase has no notion of an administrator to guard them with, and
shipping a kill switch anyone can pull is worse than not shipping one.
X_ENABLED and X_PRIMARY_PROVIDER cover the two that matter without a code
deploy. /x/status is the read-only half and is where those buttons go.

Also honest about one limit: both unofficial providers keep their own logged-in
sessions by default, so the pool has nothing to hand them unless the deployment
exposes a per-request parameter. Nothing here guesses at that parameter's name —
guessing would mean putting a live cookie on a query string somebody's instance
might log.

Verification

• 64 new tests; all 12 packages green (fail 0) under Node 24 and Node 22 (CI's version).
• pnpm build compiles; every new route registers.
• Migration applied to a scratch database: columns, tables and indexes created,
  and the backfill checked against the URL shapes the bulk import actually
  produced — including that the canonical row wins a duplicate regardless of
  insert order.
• End-to-end against a seeded database: pages, all four formats, 304 on a
  matching ETag and 200 on a stale one, canonical tags pointing at /r/ and
  /x/, and no provider name anywhere in a public document.

X_ENABLED defaults to false, so merging this collects nothing until it is
switched on.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q6QEgpuS4MLamogXtr2ZX6