sembr 1.4.0 — structured extraction (map-reduce)

Added

• LLM structured extraction (map-reduce) — opt-in per intent. When enabled, the digest pipeline first maps each recalled article into a structured, spec-defined record (source org, thesis, per-section claims with verbatim quotes), caches it, then reduces those facts into the digest instead of the raw article bodies — sharply cutting unsupported claims and mis-attribution. In A/B testing against the raw-body path on real digests, unsupported claims were roughly halved and citation precision improved. It falls back to the raw-body path automatically when extraction is unavailable, and tags each digest with a reduce_mode badge (raw / facts / facts_partial / facts_fallback_raw).
  • Per-digest source extraction — a "sources extraction" action on any history digest extracts and caches every cited article's structured record for inline inspection; per-article failures are listed individually.
  • Auto-generated, editable extraction specs — a meta-LLM drafts a per-intent extraction spec from the intent's analysis template; auto-generate / edit / validate / save in the dashboard's Advanced panel.
• Better source attribution for Wisburg reports — Wisburg report details now carry a per-article publisher/provenance, folded into the article body so extraction attributes the real institution (e.g. 花旗) to source_org instead of the generic feed label.

Fixed

• Empty LLM completions are retried instead of dropped — some providers occasionally return HTTP 200 with empty/null content under JSON mode; the chat call now retries within its existing backoff, so a single empty reply no longer drops a whole article from an extraction run.
• Feed config edits now reach the running scheduler — editing a feed's config (e.g. toggling ignore_published_watermark) while it stayed enabled left the running poll job on the old config; the PATCH handler now re-registers the job on a config change.

Upgrading

No new dependencies. One additive column (summary_history.reduce_mode) is created by an idempotent startup migration — a normal rebuild is enough:

git pull && docker compose up -d --build

The feature is off by default for every intent — nothing changes until you enable it. To turn it on: open an intent in the dashboard, generate/edit its extraction spec in the Advanced panel, then toggle extraction on. Two optional models default to your summarization model:

REDUCE_MODEL=...           # structured extraction + reduce
META_EXTRACTION_MODEL=...  # spec auto-generation

Full changelog: v1.3.0...v1.4.0