Sources in, feeds out. An open platform for databases that only ever grow: game releases, package registries, government filings, and any other public data with a big seed and daily growth. One Postgres, one Redis, and every row reachable by web, RSS, JSON, API, CLI and MCP.
The first deployment is nichedb.dev. Run your own on anything that has Postgres and Redis.
| Thing | What it is |
|---|---|
| collection | A niche: games, packages, filings. |
| source | One adapter pointed at one upstream, fetched on its own cadence. Steam's new-releases list. The npm changes feed. EDGAR Form D filings. |
| item | One row a source produced. Title, URL, when (with time_known and precision), tags, and the adapter's payload in data. |
| feed | A saved query over a collection. Has a page, RSS and JSON Feed renderings, an API endpoint, and followers who are told when it changes by push, email or signed webhook. |
Adapters are one file each in packages/adapters/src. Twenty-nine ship today across thirteen collections:
| Collection | Adapters | Key needed |
|---|---|---|
| games | steam, steam-news, igdb |
IGDB only (Twitch client) |
| packages | npm, pypi, crates, go-modules, huggingface, github-releases |
no (GitHub token optional) |
| filings | edgar, federal-register, courtlistener |
CourtListener only |
| music | musicbrainz |
no |
| books | openlibrary |
no |
| tabletop | scryfall-sets, scryfall-cards |
no |
| space | launch-library |
no |
| chess | lichess-broadcasts |
no |
| alerts | usgs-earthquakes, nws-alerts, gdacs |
no |
| outages | statuspage (any Statuspage host) |
no |
| extensions | firefox-addons, vscode-extensions, mcp-registry |
no |
| health | openfda-recalls, clinical-trials |
no |
| research | arxiv, crossref |
no |
After ingest, every item is enriched by the enrichers that apply to it (packages/enrichers/src), and the results live on the item under enrichment.<name>:
| Enricher | Adds | Default on for |
|---|---|---|
youtube |
top videos (trailers, official audio, webcasts); Data API key optional | games, music, tabletop, space, chess, books |
wikipedia |
the article's lead paragraph and picture | games, music, books, space, tabletop |
github-repo |
stars, forks, topics, licence, language, last push | packages, extensions |
npm-stats |
last week's downloads | packages |
sec-company |
tickers, exchange, industry, state, website of the filer | filings |
semantic-scholar |
TL;DR, citation counts, open-access PDF (key optional) | research |
openlibrary-work |
description and subjects | books |
opengraph |
the page's own preview image and description | most collections |
A feed's enrichers list picks which of these it shows; absent means the collection's defaults. The feed builder exposes them as checkboxes. Items are enriched once, newest first with a fair share per collection, and a missing image, summary or tags are filled from whatever the enrichers found while the source's own words always win.
git clone https://github.com/profullstack/niche-db && cd niche-db
bun install
docker compose up -d db redis # or point at your own Postgres 16+ and Redis
cp .env.example .env # DATABASE_URL, SITE_URL, CONTACT_EMAIL at minimum
bun run build:client
bun run dev # web + worker in one process on :3000Migrations apply themselves on boot. The three collections, their default sources and a dozen feeds are seeded on first boot; sources whose adapter needs a credential the deployment lacks are created paused. The first account to sign in is an admin.
bun run ingest [slug ...] runs sources from a terminal without Redis. bun test runs the suite against an in-process Postgres (PGlite), so it needs no server.
- Web: a PWA.
/sourcesmanages sources (status, last run, next run, run now, pause, edit config)./feeds/newbuilds a feed with a live preview./settingsholds passkeys, API keys and notifications. - RSS / JSON Feed:
/f/<slug>.rss,/f/<slug>.json. - API:
/api/v1, documented at/docs/api. Reads need no key. - CLI:
npm i -g @profullstack/nichedb→nichedb. Documented at/docs/cli. - MCP:
/mcp(Streamable HTTP, stateless) ornichedb mcpover stdio. Documented at/docs/mcp. - llms.txt:
/llms.txtdescribes the whole deployment in one document.
Magic link and passkey only; there is no password column. API keys are minted from settings and shown once.
Free pages and feeds carry one CrawlProof ad and a tracker (CRAWLPROOF_AD_SLOT, CRAWLPROOF_SITE_ID). Pro ($120 a month through CoinPay) is the tier without them: no ads, no tracking, the high API rate limit, unlimited feeds, own sources, and a crawl pass for the whole term (GET /api/v1/crawl-pass) so a member's own agents walk through the paywall. Referral links give the new customer 20% off and the referrer 60% of the first payment.
Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent…) get a 402 with an x402 offer and buy a pass at /crawl: $1 a day for everything, and the more a buyer has paid here the less a day costs (CRAWL_LOYALTY, default 20% off after $10, 40% after $50, 60% after $100; every sale is a row in crawl_sales). A pass may switch ads or tracking off for its own requests with ?disable=ads,tracking. People, search engines and retrieval crawlers pass through untouched.
Bun, Hono (server-rendered JSX), Postgres via Bun's native driver, BullMQ on Redis, Biome. bun apps/worker/src/enrich-cli.js [n] runs enrichment from a terminal. Shared Profullstack modules: @profullstack/x402-gateway, @profullstack/emailer, @profullstack/coinpay, @profullstack/referrals, @profullstack/api-key-manager, @profullstack/autoblog (signed webhooks), @profullstack/favicon-generator.
Everything is read once from the environment in packages/config/src/index.js; .env.example lists every variable. Production secrets live in the nichedb--prod logicsrc vault:
bun run secrets:pull # logicsrc teams pull profullstack nichedb prod --env .env
bun run secrets:push # after adding a key locallyROLES=web,worker runs everything in one container; ROLES=worker on a second service splits ingestion out when one instance stops being enough.
One Dockerfile, one Railway service, plus Railway's Postgres and Redis. railway.json sets the healthcheck at /healthz. Set SITE_URL to the public origin before anyone registers a passkey: the passkey relying-party id is derived from it.
import { defineAdapter } from '@nichedb/core/adapter';
export const example = defineAdapter({
name: 'example',
title: 'Example API',
collection: 'packages',
description: 'What it watches, in one paragraph.',
kinds: ['thing'],
cadenceMinutes: 30,
configFields: [{ key: 'topic', label: 'Topic', required: true }],
defaultSources: [{ slug: 'example-default', name: 'Example', config: { topic: 'all' } }],
async pull({ config, cursor, http, log, budget, deadline }) {
const page = await http.json(`https://example.com/api?topic=${config.topic}&since=${cursor.since ?? ''}`);
return {
items: page.results.map((r) => ({ externalId: r.id, kind: 'thing', title: r.name, url: r.url, publishedAt: r.at, tags: r.tags, data: r })),
cursor: { since: page.next },
note: `${page.results.length} things`,
};
},
});Register it in packages/adapters/src/index.js. The core normalises items, hashes them so unchanged rows cost no write, writes in batches, records the run and reschedules.
MIT