Repository navigation
Home
Slackpipe is a local-first, multi-workspace Slack archival pipeline built around Slackdump, DuckDB, and Dagster. It preserves durable raw archives, converts coherent snapshots into a canonical relational model, verifies source/canonical equivalence, and exposes operational health through Dagster checks and Prometheus metrics.
- Architecture — boundaries, data flow, and storage contracts
- Workspace-Configuration — credential blocks and workspace discovery
- Extraction-and-Attachments — Slackdump planning, locks, and files
- Canonical-DuckDB — schema, transactions, lineage, and verification
- Dagster-Orchestration — assets, checks, jobs, schedules, and sensors
- Compaction-and-Recovery — proof-gated maintenance and backups
- Deployment-and-Operations — Compose deployment and runbook
- Testing-and-Security — evidence, threat boundaries, and limitations
Slackquery turns Slackpipe's canonical DuckDB and attachment tree into immutable hybrid-search artifacts and serves them through MCP. Slackpipe remains the sole canonical writer; Slackquery receives source data read-only.
- Workspace credentials stay outside Git and orchestration metadata.
- Raw Slackdump mutation is serialized.
- Live SQLite archives are snapshotted through SQLite's backup API.
- Canonical DuckDB updates are transactional and single-writer.
- Source continuity and key-set invariants are validated before trust.
- Downstream consumers do not write the canonical database.
Slackpipe can preserve only data visible to the supplied Slack credentials and returned by Slack/Slackdump. A successful run verifies pipeline consistency; it does not prove that Slack exposed every historical object.