Architecture

Architecture

Ownership boundaries

flowchart TB
    subgraph Secrets[Secret boundary]
      W[Workspace credential blocks]
    end
    subgraph Raw[Raw archive boundary]
      SD[Slackdump]
      SQL[(Workspace SQLite archives)]
      FILES[(Attachment tree)]
    end
    subgraph Canonical[Canonical boundary]
      SNAP[Coherent SQLite snapshot]
      TX[Validation + transaction]
      DB[(Canonical DuckDB)]
    end
    subgraph Control[Control plane]
      DG[Dagster]
      PG[Pushgateway]
    end
    subgraph Search[Independent search boundary]
      SQ[Slackquery]
    end

    W --> SD --> SQL
    SD --> FILES
    SQL --> SNAP --> TX --> DB
    DG --> SD
    DG --> TX
    TX --> PG
    DB -->|read-only| SQ
    FILES -->|read-only| SQ

Loading

Slackpipe owns credentials, extraction, raw persistence, canonicalization, lineage, and validation. Dagster owns orchestration state, not source truth. Slackquery is independently deployable and never writes Slackpipe data.

Data lifecycle

1. Parse and validate workspace blocks.
2. Select initial archive or incremental resume behavior.
3. Execute Slackdump under a global advisory lock.
4. Reconcile attachment metadata and local object paths.
5. Create a coherent SQLite backup snapshot.
6. Validate source schema, continuity, and completeness.
7. Upsert canonical tables inside one DuckDB transaction.
8. Prove source/canonical invariants.
9. Commit checkpoints and publish latest-run metrics.

Concurrency model

• one global Slackdump mutation lock;
• one canonical DuckDB writer lock;
• Dagster pool and run concurrency limits;
• serialized rollout stages where source dependencies require ordering;
• immutable or read-only access for downstream consumers.

Failure boundary

The raw archive remains the rebuildable source. A failed canonical transform rolls back its transaction. A failed check prevents operators from treating the new state as complete. Existing canonical rows remain available for readers.