Nexum

Nexum (Latin: a binding, a tie) — the connections between ingestion, inference, evaluation, and retraining.

A real-time trend and content-intelligence system that drinks the Bluesky firehose, detects hashtag bursts as they happen, and (phase 2) classifies posts in-stream with a small language model distilled from a cloud teacher — teacher-shadowed, registry-versioned, and retrained from its own disagreement data. Runs on one small VPS.

The problem it exists to solve: everyone's demo calls a cloud LLM on a webhook. Nexum replaces the API model in the hot path with a self-distilled ~270M model at measured teacher agreement, ~0 marginal cost, and a measured p99 — on a $10 box.

What it does (MVP, phase 1)

• Ingester — asyncio + websockets consumer of the Jetstream firehose; time-cursor resume with rewind (at-least-once); produces raw events to Redpanda via confluent-kafka with delivery callbacks and backpressure.
• Burst detector — Quix Streams dataflow: posts exploded by hashtag, hopping-window counts with RocksDB-backed state, z-score against a rolling EMA baseline; bursts go to a bursts topic and Discord.
• Archiver — consumer that batches posts into date-partitioned Parquet (the future training corpus and the DuckDB/dbt batch leg).
• Live dashboard — FastAPI + SSE: live throughput, top hashtags, burst feed.

Architecture

flowchart LR
    JS[Jetstream wss] --> ING[ingester\nasyncio + confluent-kafka]
    ING --> RP[(Redpanda\nposts.raw)]
    RP --> DET[burst detector\nQuix Streams windows + state]
    RP --> ARC[archiver\nParquet -> DuckDB/dbt]
    RP --> DASH[dashboard\nFastAPI SSE]
    DET --> BT[(bursts topic)]
    BT --> DASH
    DET --> DISC[Discord alerts]

Loading

Phases 2–3 add: llama.cpp serving a QLoRA-distilled GGUF classifier in the hot path with sampling, 1% teacher shadowing for drift, dbt marts over the archive, and nightly disagreement-mined retraining behind an MLflow eval gate with adapter hot-swap + rollback. Phase 4 (optional): port one dataflow to PyFlink and write the comparison.

How to run

uv sync                      # deps (Python 3.12+)
docker compose up -d         # single-node Redpanda on localhost:19092 (+ console on :8080)
uv run python -m nexum.ingester    # terminal 1 — drinks the firehose
uv run python -m nexum.detector    # terminal 2 — windows + z-score bursts
uv run python -m nexum.archiver    # terminal 3 — Parquet archive
uv run python -m nexum.dashboard   # terminal 4 — http://localhost:8765

Configuration is env-only, prefixed NEXUM_ (see src/nexum/config.py for every knob and its default). The ones you'll actually set:

NEXUM_BROKER=localhost:19092
NEXUM_DISCORD_WEBHOOK=   # empty -> alerts log to stdout instead
NEXUM_Z_THRESHOLD=3.0
NEXUM_MIN_COUNT=10

Tests and gates: uv run pytest, uv run ruff check ., uv run mypy src/ — same as CI.

Deploy: deploy/*.service systemd units; /etc/nexum.env for config. Runbook lands at first deploy.

Assumptions (scope and non-goals)

• One VPS. Single-node Redpanda; no replication story, on purpose (ADR-003).
• Burst key = hashtag in the MVP. "Topic" means the phase-2 SLM label, nothing else.
• Classification is title/text-level, one topic label per post, small fixed taxonomy (exact label list picked from the real corpus at phase-2 start). Language comes from the event's native langs field — no model needed.
• Sampling under load is expected behavior, not a bug: 1-in-N baseline + 100% of burst candidates (phase 2).
• Creates only. Deletes/updates are not ingested; the archive is a research corpus, not a mirror.
• DuckDB for the batch leg (free, on-box); Parquet keeps a BigQuery swap cheap if ever wanted. No Redis until something actually needs a hot store.
• Flink deliberately deferred to phase 4 (ADR-002).
• The MVP ingests for a real week with measured — not projected — numbers before anything lands on a resume. See docs/metrics.md for the contract.

Decisions

docs/adr/ — Python end-to-end (001), Quix Streams (002), Redpanda (003), Jetstream (004). Domain vocabulary: CONTEXT.md.