Canonical DuckDB

Canonical DuckDB

Tables

The canonical schema includes:

• schema_metadata
• ingestion_runs
• source_archives
• ingestion_checkpoints
• workspaces
• channels
• users
• channel_members
• messages
• reactions
• files
• message_events
• attachment_backfills
• document_chunks

Transaction model

Transformation obtains the canonical writer lock, opens DuckDB, creates or validates schema, and applies one source snapshot inside a transaction. The checkpoint and lineage records advance only with the associated canonical data.

Idempotence and newest-row semantics

Stable composite keys prevent duplicate canonical entities. Mutable Slack rows are upserted using source revision and payload information. Reprocessing an unchanged coherent source is idempotent.

Lineage

Slackpipe records source archive identity, source schema versions, payload fingerprints, chunk/high-water information, run status, and checkpoint state. Unexpected source replacement, truncation, or lineage regression is rejected unless an explicit bootstrap/recovery path is selected.

Verification

Source-versus-canonical validation checks relevant combinations of:

• key-set equality;
• row counts and newest-row maps;
• payload/hash agreement;
• latest timestamps;
• message/thread relationships;
• reaction and file references;
• attachment path/byte coverage;
• checkpoint and source identity continuity.

Validation failures are operational failures, not warnings to ignore.