sembr 1.5.0 — per-intent knowledge base, permanent news archive, unified news search

Added

• Per-intent knowledge base (KB) — opt-in per intent (kb_enabled, default off). Each enabled intent maintains a markdown event index (events.md): one entry per event key with its first-seen date, latest date, and current status, grouped into topical sections. After every cron digest the new facts are incrementally merged into that index — never re-distilled from scratch — so the file converges instead of growing without bound, and each day's change is a reviewable git diff. A weekly lint pass checks for contradictions, stale entries, and orphans, applies its fixes, and commits them; git history is the rollback path. Edit any KB from the Intents tab: an inline modal with a highlighted editor and markdown preview, openable full-screen; Build / Rebuild takes a days lookback.
  • This release produces and maintains the index only. The digest's {history} slot is unchanged, so a KB-enabled intent emits exactly the same digest as before. Feeding the index back into the reduce step — and with it the incremental-label ([新增] / [持续] / [升级] / [降级]) accuracy that motivated the KB in 1.4.0's roadmap — is the next step, not this one.
• Permanent news archive — the retention job now moves expired articles into a second collection (news_archive) with their vectors instead of deleting them, so news that has aged out of the live window stays semantically searchable indefinitely. A failed archive write aborts the run with nothing deleted, so an article can never end up in neither store. QDRANT_ARCHIVE_ENABLED=false reverts to plain deletion.
• Unified news search — POST /api/news/search — one endpoint over the whole news timeline. The split between the live store and the archive is an internal detail: the endpoint queries both and returns a single ranked list, with no scope parameter and no marker on a hit. Semantic mode (query, deepen with exclude_ids) and newest-first filter listing (cursor pagination) share one filter schema — ingestion and publication time windows, feed include/exclude, title keyword, URL domain, minimum body length, previously matched intent, and language. A failure in either store fails the whole request rather than returning half the timeline with a 200.
  • Four derived filter fields (published_at_ts, body_len, lang, url_domain) are written from one shared definition by every path that stores an article, so a filter means the same thing at both ends of the timeline. A self-healing background job backfills them onto points written before this release; while that queue is non-empty, filters that depend on them return a warning, and the warning is explicit that the gap is in recent coverage, not the deep archive.
  • GET /api/dashboard/maintenance/qdrant_stats — operator view: per-store point counts, ingestion time ranges, alias health, and backfill queue depth. Unlike search, it does expose the storage split.

Fixed

• newsapi feeds no longer burn their whole quota after an outage — when a fetch hit the page cap, the feed's cursor advanced to the oldest article in the batch, on the theory that the next tick would resume there and walk back through the backlog. It never converges: the API returns newest-first, so every tick re-fetched the same most-recent window and its oldest article always sat a fixed number of articles behind now() — the cursor tracked now() instead of climbing toward it, the watermark could never fire again, and each tick spent the full page cap forever. Latent until an outage long enough to push the cursor past the fetched window's span, then permanent. The cursor now advances to the newest article in the batch, so the watermark fires again within a page or two and cost returns to normal.

Upgrading

No new dependencies and no schema migration beyond an additive intents.kb_enabled column created at startup. A normal rebuild is enough:

git pull && docker compose up -d --build

The archive is on by default. After this release the retention job stops deleting and starts moving, so expect news_archive to begin growing. Archived articles exist only there — dropping that collection loses them permanently. To keep the old behaviour:

QDRANT_ARCHIVE_ENABLED=false

KB is off by default for every intent; enable it per intent in the dashboard. Two optional models default to your summarization model:

KB_MERGE_MODEL=...    # incremental merge, runs after every digest (flash tier is fine)
KB_DISTILL_MODEL=...  # one-off "rebuild KB" (a stronger model is worth it — it seeds every later merge)

The derived-field backfill starts itself two minutes after boot and repeats every 30 minutes until the queue is empty. On an existing deployment, expect a warning on language / url-domain / body-length / publication-time filters until it converges — on a 103k-point store it took under two minutes.

POST /api/archive/search and GET /api/archive/stats were developed and replaced within this release cycle and never shipped in a tagged version, so there is nothing to migrate: agents should call POST /api/news/search.

Full changelog: v1.4.0...v1.5.0