Sift 0.1.0

First tagged release. Clone it and it runs — no database to provision, no key to obtain, no native module to compile.

git clone https://github.com/micaho26/sift && cd sift
pnpm install     # 7.8s on a clean clone
pnpm seed        # optional: 37 realistic items to judge it before configuring
pnpm dev

Requires Node 24+ and pnpm. Nothing else.

What's in it

Collects — 8 keyless connectors (Hacker News, arXiv, GitHub, Hugging Face, Reddit, RSS and more) plus a Manifest V3 browser extension that captures from X and 小红书 by reading the responses those apps already fetch for themselves. No scraping of rendered DOM, no login automation.

Ranks — six explainable components: velocity, authority, relevance, novelty, depth, recency. Per-source counter weighting, because a bookmark on X and an upvote on HN do not mean the same thing. Engagement-bait penalty. Every score opens to its full breakdown; nothing is a black box.

Dedupes — two tiers. SimHash with LSH banding for reposts and near-identical text; embedding similarity, guarded by a shared specific entity and a time window, for the same story retold in different words. Both thresholds were set by measuring real pairs, not guessed: a genuine repost sits at Hamming 22 while unrelated items sit at 27–31, which is why the second tier exists at all.

Searches — BM25, concept expansion and vector KNN, fused by Reciprocal Rank Fusion, diversified by MMR. Under 15ms, and exact rather than approximate KNN, because a personal corpus is small enough that approximation buys nothing and costs recall. Chinese works as well as English: unicode61 has no word segmentation and treats a whole Han run as one token, so documents carry a parallel bigram column and queries are windowed and affix-stripped.

Synthesises — cited briefings and retrieval-grounded chat over your own corpus. Optional, and the only part that can talk to a model provider. Without a key, everything else still works.

Zero native dependencies

SQLite is Node 24's built-in node:sqlite. There is no node-gyp step, no prebuild download, no "works on my machine" — pnpm install cannot fail on a compile step. The embedder that ships is a deterministic hashing embedder with no model download; a provider key upgrades it if you want one.

Local means local

One SQLite file on your machine. No account, no telemetry, no data leaving the box. API keys, if you add them, are write-only over the API — GET /settings reports apiKeySet: true and never the key.

Verified, not asserted

75 tests across core, server and the extension. The server tests run against a real node:sqlite file rather than a mock, because every bug that layer actually produced was a SQL-level one. Scoring tests assert behaviour — "a substantive thread outranks viral bait", "clock skew cannot push a score out of range" — not magic numbers. The website makes no third-party requests at all; the fonts are self-hosted, which is the only way the no-telemetry claim is true of the marketing page too.

Links

Website · Screenshot tour · Architecture · Scoring model · Extension · API

MIT. The ideas are not new — RRF is Cormack et al. (SIGIR 2009), SimHash is Charikar (2002) with Manku et al.'s band scheme, MMR is Carbonell & Goldstein (1998), the velocity gravity term is Hacker News's. The implementation is.