Compaction and Recovery

Compaction and Recovery

Offline compaction

Raw Slackdump archives accumulate historical row revisions. Slackpipe can build a compacted SQLite archive containing newest semantic rows while preserving the tables and metadata needed for future resumes.

Compaction writes a new destination. It never edits the active source in place.

Proof gate

Before replacement, verification compares source and compacted archives using newest-row maps and semantic counts. The compacted file must pass SQLite integrity checks and Slackpipe's equivalence rules.

Checkpoint rebasing

Compaction changes source file identity and physical layout. After a verified replacement, Slackpipe rebases canonical checkpoints to the compacted source so future resumes do not treat maintenance as unapproved source replacement.

Recovery sequence

1. Stop affected schedules and confirm no active writer.
2. Preserve raw archive, attachments, canonical DuckDB, and Dagster PostgreSQL.
3. Restore a mutually consistent set.
4. Run archive and source/canonical verification.
5. Reload the code location.
6. Launch one bounded reconciliation.
7. Resume schedules only after checks pass.

Backup scope

Back up raw archives, attachments, canonical DuckDB after checkpointing, Dagster PostgreSQL, and secret configuration through a secure system. Do not put any of those artifacts in the Git repository.