sembr 1.5.0 — per-intent knowledge base, permanent news archive, unified news search

Added

• Per-intent knowledge base (KB) — opt-in per intent (kb_enabled, default off). Each enabled intent maintains a markdown event index (events.md): one entry per event key with its first-seen date, latest date, and current status, grouped into topical sections. After every cron digest the new facts are incrementally merged into that index — never re-distilled from scratch — so the file converges instead of growing without bound, and each day's change is a reviewable git diff. A weekly lint pass checks for contradictions, stale entries, and orphans, applies its fixes, and commits them; git history is the rollback path. Edit any KB from the Intents tab: an inline modal with a highlighted editor and markdown preview, openable full-screen; Build / Rebuild takes a days lookback.
  • This release produces and maintains the index only. The digest's {history} slot is unchanged, so a KB-enabled intent emits exactly the same digest as before. Feeding the index back into the reduce step — and with it the incremental-label ([新增] / [持续] / [升级] / [降级]) accuracy that motivated the KB in 1.4.0's roadmap — is the next step, not this one.
• Permanent news archive — the retention job now moves expired articles into a second collection (news_archive) with their vectors instead of deleting them, so news that has aged out of the live window stays semantically searchable indefinitely. A failed archive write aborts the run with nothing deleted, so an article can never end up in neither store. QDRANT_ARCHIVE_ENABLED=false reverts to plain deletion.
• Unified news search — POST /api/news/search — one endpoint over the whole news timeline. The split between the live store and the archive is an internal detail: the endpoint queries both and returns a single ranked list, with no scope parameter and no marker on a hit. Semantic mode (query, deepen with exclude_ids) and newest-first filter listing (cursor pagination) share one filter schema — ingestion and publication time windows, feed include/exclude, title keyword, URL domain, minimum body length, previously matched intent, and language. A failure in either store fails the whole request rather than returning half the timeline with a 200.
  • Four derived filter fields (published_at_ts, body_len, lang, url_domain) are written from one shared definition by every path that stores an article, so a filter means the same thing at both ends of the timeline. A self-healing background job backfills them onto points written before this release; while that queue is non-empty, filters that depend on them return a warning, and the warning is explicit that the gap is in recent coverage, not the deep archive.
  • GET /api/dashboard/maintenance/qdrant_stats — operator view: per-store point counts, ingestion time ranges, alias health, and backfill queue depth. Unlike search, it does expose the storage split.

Fixed

• newsapi feeds no longer burn their whole quota after an outage — when a fetch hit the page cap, the feed's cursor advanced to the oldest article in the batch, on the theory that the next tick would resume there and walk back through the backlog. It never converges: the API returns newest-first, so every tick re-fetched the same most-recent window and its oldest article always sat a fixed number of articles behind now() — the cursor tracked now() instead of climbing toward it, the watermark could never fire again, and each tick spent the full page cap forever. Latent until an outage long enough to push the cursor past the fetched window's span, then permanent. The cursor now advances to the newest article in the batch, so the watermark fires again within a page or two and cost returns to normal.

Upgrading

No new dependencies and no schema migration beyond an additive intents.kb_enabled column created at startup. A normal rebuild is enough:

git pull && docker compose up -d --build

The archive is on by default. After this release the retention job stops deleting and starts moving, so expect news_archive to begin growing. Archived articles exist only there — dropping that collection loses them permanently. To keep the old behaviour:

QDRANT_ARCHIVE_ENABLED=false

KB is off by default for every intent; enable it per intent in the dashboard. Two optional models default to your summarization model:

KB_MERGE_MODEL=...    # incremental merge, runs after every digest (flash tier is fine)
KB_DISTILL_MODEL=...  # one-off "rebuild KB" (a stronger model is worth it — it seeds every later merge)

The derived-field backfill starts itself two minutes after boot and repeats every 30 minutes until the queue is empty. On an existing deployment, expect a warning on language / url-domain / body-length / publication-time filters until it converges — on a 103k-point store it took under two minutes.

POST /api/archive/search and GET /api/archive/stats were developed and replaced within this release cycle and never shipped in a tagged version, so there is nothing to migrate: agents should call POST /api/news/search.

Full changelog: v1.4.0...v1.5.0

sembr 1.4.0 — structured extraction (map-reduce)

Added

• LLM structured extraction (map-reduce) — opt-in per intent. When enabled, the digest pipeline first maps each recalled article into a structured, spec-defined record (source org, thesis, per-section claims with verbatim quotes), caches it, then reduces those facts into the digest instead of the raw article bodies — sharply cutting unsupported claims and mis-attribution. In A/B testing against the raw-body path on real digests, unsupported claims were roughly halved and citation precision improved. It falls back to the raw-body path automatically when extraction is unavailable, and tags each digest with a reduce_mode badge (raw / facts / facts_partial / facts_fallback_raw).
  • Per-digest source extraction — a "sources extraction" action on any history digest extracts and caches every cited article's structured record for inline inspection; per-article failures are listed individually.
  • Auto-generated, editable extraction specs — a meta-LLM drafts a per-intent extraction spec from the intent's analysis template; auto-generate / edit / validate / save in the dashboard's Advanced panel.
• Better source attribution for Wisburg reports — Wisburg report details now carry a per-article publisher/provenance, folded into the article body so extraction attributes the real institution (e.g. 花旗) to source_org instead of the generic feed label.

Fixed

• Empty LLM completions are retried instead of dropped — some providers occasionally return HTTP 200 with empty/null content under JSON mode; the chat call now retries within its existing backoff, so a single empty reply no longer drops a whole article from an extraction run.
• Feed config edits now reach the running scheduler — editing a feed's config (e.g. toggling ignore_published_watermark) while it stayed enabled left the running poll job on the old config; the PATCH handler now re-registers the job on a config change.

Upgrading

No new dependencies. One additive column (summary_history.reduce_mode) is created by an idempotent startup migration — a normal rebuild is enough:

git pull && docker compose up -d --build

The feature is off by default for every intent — nothing changes until you enable it. To turn it on: open an intent in the dashboard, generate/edit its extraction spec in the Advanced panel, then toggle extraction on. Two optional models default to your summarization model:

REDUCE_MODEL=...           # structured extraction + reduce
META_EXTRACTION_MODEL=...  # spec auto-generation

Full changelog: v1.3.0...v1.4.0

sembr 1.3.0 — Wisburg research-note source

Added

• Wisburg source — new wisburg-report feed type pulling Wisburg's open-API research-note streams: institutional research notes (/api/reports), earnings-call transcripts (/api/earningscalls), and asset-management reports (/api/am-reports). One feed per stream — pick the endpoint URL from the datalist in the create-feed form; articles carry Wisburg's pre-digested markdown summary as the body, so they flow straight into embedding / intent matching / digests. Incremental sync starts from the last 24 h on first pull (no historical backfill), with a 1 h overlap window and a 7 d clamp for stale cursors.

Upgrading

No new dependencies and no schema migration — a normal rebuild is enough:

git pull && docker compose up -d --build

To enable Wisburg feeds, add your key (from the Wisburg developer console) to .env:

WISBURG_API_KEY=sk-...

Leave it unset and Wisburg feeds simply stay inactive; everything else is unchanged.

Full changelog: v1.2.0...v1.3.0

sembr 1.2.0 — Email PDF digest attachment

Added

• Email PDF attachment — opt-in per-intent PDF copy of each digest, attached to the notification email. Rendered by WeasyPrint from a dedicated A4 print template (clickable source URLs; CJK fonts bundled in the Docker image). Enable it with the new Attach PDF toggle in the intent editor's Channels · Email section — off by default, so existing intents are unchanged.

Fixed

• NewsAPI cap_reached now dispatches the already-fetched batch and advances each feed's cursor to the oldest article instead of dropping the entire tick — fixing an infinite retry loop on deep backlogs.
• Email digest tables now render with visible borders in Outlook and Gmail.

Upgrading

This release adds WeasyPrint and its native libraries (Pango/Cairo + fonts-noto-cjk) to the image, so rebuild rather than just restart:

git pull && docker compose up -d --build

No config migration needed — attach_pdf defaults off.

Full changelog: v1.1.0...v1.2.0

sembr 1.1.0 — History, backfill, and aggregate analysis

The intent radar gets a memory. Cron match summaries are now persisted,
past fire-times can be replayed through backfill, and multiple history
rows can be aggregated through a single LLM call — giving each digest
the context of what came before.

What's new

• History persistence — every cron digest is saved to SQLite. Browse,
  delete, export. Survives restarts.
• Backfill — replay past fire-times through the scan pipeline without
  waiting for the next scheduled tick.
• {history} in prompts — inject recent summaries into the summarizer
  so the LLM builds on prior analysis rather than starting from zero each
  time.
• Aggregate & export — run one LLM call over weeks of history, or
  export the raw data as JSON.
• 7 new API endpoints — history CRUD, backfill+status, aggregate
  preview/send, export.
• Frontend UI — history table with timezone-aware timestamps, backfill
  button, Summarize + Export modals, Markdown rendering with XSS protection.

See CHANGELOG.md for the full list.

sembr 1.0.0 — initial public release

sembr 1.0.0 — Reverse RAG, self-hosted.

sembr is a self-hosted intent radar. You describe what you care about once — "monitor Fed policy impact on emerging-market currencies" — and it continuously scans RSS feeds, news APIs, and social streams, matches articles to your intent via semantic vectors, and delivers LLM-analyzed digests from whatever angle you configure.

Apache-2.0. ~1 GB total image footprint. Runs on a 4 GB box.

git clone https://github.com/Peakstone-Labs/sembr.git
cd sembr && cp .env.example .env   # set EMBEDDER_API_KEY + SMTP creds
docker compose up --build

Or — paste this to your AI coding agent:

Read https://github.com/Peakstone-Labs/sembr/blob/main/agent/INSTALL.md and follow it to install sembr on this machine.

15 minutes from git clone to your first email digest. Live demo: panel.peakstone-labs.com/#news — what sembr is producing today.

---

What's in the box

• Reverse-RAG matcher — cron + event modes; per-intent threshold (0.60–0.95); two-layer dedup (MD5 ingest + per-intent match_seen).
• 53 pre-loaded sources across three source types — 22 RSS feeds (Guardian / SCMP / NPR / Bloomberg Markets / 华尔街见闻 / 第一财经 / 36氪 / Nature ×3 / HelloGitHub / …), 30 NewsAPI.ai aggregator sources (Reuters / BBC / NYT / WSJ / FT / Economist / TechCrunch / Wired / …), 1 Twitter feed via bundled RSSHub sidecar.
• BGE-M3 embeddings on SiliconFlow — 1024-dim, 8192-token context, native bilingual EN/CN. Free at any volume on the BGE-M3 tier.
• DeepSeek-V4-Flash summaries — 1M-token context lets one digest ingest a hundred long articles for well under a cent.
• Custom prompt templates — system + instruction templates with strict-placeholder validation; per-intent template selection; edits take effect on the next scheduler tick.
• Email digest channel — SMTP multipart/related, inline assets, matcher-score badges, rendered in each intent's own timezone. Telegram / Discord / Slack scaffolded for post-1.0.
• Qdrant 1.17.1 dual-collection design (intents + news) with scalar int8 quantization, always_ram=True ANN over disk-resident raw vectors.

Built for agents

sembr was designed agent-first from day one:

• agent/INSTALL.md — a 6-phase install protocol written for AI coding agents to read. Paste the URL to Claude Code / Cursor / Cline / Aider / Continue / Roo and walk away for 15 minutes.
• agent/sembr/ — a full Agent Skills bundle (SKILL.md + endpoint / schema / recipe / error references). Drop it into ~/.claude/skills/sembr/ and your agent can drive a running sembr instance: create intents, tune thresholds, edit templates, trigger diagnostic fires.
• POST /api/external/intents/{id}/fire — synchronous endpoint returning matched articles + LLM summary in one round-trip. No notification side-effects, no match_seen writes, per-call overrides for lookback / threshold / feed scope. Hermes / OpenClaw / LangGraph / your own orchestrator can treat sembr as a tool node.

Deploy sembr with an agent. Drive it as an agent's tool. Both work out of the box.

Why "Reverse RAG"?

In classic RAG, you ask a question; the system retrieves and answers.

In Reverse RAG, you describe a long-running intent once; the system retrieves forever, scanning incoming articles against your intent vector on a schedule you set.

Attention Is All You Need — AI is your attention.

The matcher is built literally on attention-based BGE-M3 embeddings. Out the back: the digest comes through whatever framing you wrote into the prompt template. Same engine, different angle per intent.

Live demo

panel.peakstone-labs.com/#news — two production intents (霍尔木兹海峡危机日报 / AI 算力日报) running on sembr right now, regenerated daily, anyone can read. What's on screen is what sembr produces.

Right-sized

Component	Memory (default 53 sources)	mem_limit (4× headroom)
api	~125 MiB	1.5G
qdrant	~520 MiB	2G
rsshub	~355 MiB	512M
Total	~1 GB	—

Homelab / Mac mini / NAS / $10 VPS all comfortable. First-pull total ~1 GB (Python 3.12 base + Qdrant + RSSHub).

Security model

• DashboardTokenMiddleware gates /api/* behind X-Dashboard-Token. /health is intentionally auth-free.
• sembr logs an ERROR if DASHBOARD_TOKEN is empty — OK for localhost dev, unsafe for any host reachable beyond loopback.
• See docs/deployment/public.md for the end-to-end public-deployment guide: reverse-proxy samples (Caddy / nginx / Cloudflare Tunnel), ufw, SSH hardening, nmap verification.
• Private Vulnerability Reporting enabled — please report security issues via the Security tab, not public issues. See SECURITY.md.

Tech stack

Python 3.12 · FastAPI 0.115 · Pydantic v2 · APScheduler 3.11.2 · Qdrant 1.17.1 · aiosqlite 0.20 · httpx 0.27 · feedparser 6 · BGE-M3 (SiliconFlow) · DeepSeek-V4-Flash (SiliconFlow).

Links

• Quickstart: README.md · zh-CN
• Agent install protocol: agent/INSTALL.md
• Docs: peakstone-labs.github.io/sembr
• Live demo: panel.peakstone-labs.com/#news
• Full changelog: CHANGELOG.md
• Contributing: CONTRIBUTING.md · zh-CN

Acknowledgments

Built by Peakstone Labs. sembr is our first open-source project — and not the last.

We're a quant research outfit. The problems sembr solves (information overload, unstructured-source tracking, agent-callable monitoring) are the ones we hit ourselves every day. Open-sourcing them is the cheapest way to make them better — for us and for everyone facing the same problems.

---

License: Apache-2.0 · See NOTICE for attribution.

Issues: GitHub Issues · Discussion: GitHub Discussions · Security: PVR