Add Cluster Backup Restore And Failure Runbooks

[TheSilkky]

Add Cluster Backup Restore And Failure Runbooks

Priority

P3

Type

documentation

Labels

Suggested GitHub labels:

• backlog
• docs
• deployment
• maintenance
• security

Branch scope

• Current branch: develop
• Current HEAD: 539bb8d0a2cab1b9dc5e18ff394ae9959c8d7350
• Target branch, if known: develop
• Reviewed branch/ref, if applicable: develop
• Reviewed commit SHA, if applicable: 539bb8d0a2cab1b9dc5e18ff394ae9959c8d7350
• Target release/version, if applicable: post-#88 production-cluster follow-up
• Scope classification: follow-up-after-merge
• Scope note: This draft was generated from the branch above. Revalidate against the target branch before creating or closing public GitHub issues if the branch has moved or has not yet merged.

Summary

Add operational runbooks for backup, restore, and failure handling before any production-cluster backend is recommended for real deployments.

Context

The current docs/retention-backup-deletion.md explains SQLite and local blob backup consistency. PR #88 added docs/production-cluster-scope.md, which requires future PostgreSQL and S3-compatible backends to keep metadata and blobs consistent, clean up abandoned staging state, and update deployment, backup, restore, security, and threat-model docs before recommending production cluster deployment.

Proposed change

Create cluster-specific runbook documentation that explains how operators should back up and restore metadata, encrypted blobs, configuration, and coordination state once optional cluster backends exist. Include failure-mode guidance for database/object mismatch, abandoned staging objects, partial restore, and coordination service loss.

Acceptance criteria

• Runbook states which systems are durable sources of truth and which systems are coordination-only.
• Runbook covers backup consistency between PostgreSQL metadata and S3-compatible encrypted blobs.
• Runbook covers restore validation using private/local bind addresses only.
• Runbook describes failure behavior for missing blobs, missing metadata, abandoned staging objects, and unavailable coordination service.
• Runbook preserves warnings that the project is not production-ready public infrastructure until access control, retention, observability, and abuse controls exist.
• Runbook avoids secrets, raw tokens, private deployment details, exploit details, plaintext, raw keys, and user safety data.

Tests / validation

• gofmt -w ./cmd ./internal ./migrations, if Go files change
• go test ./..., if code changes
• go vet ./..., if code changes or CI/testing changes
• simulator smoke test, if relevant
• docs updated, if relevant
• revalidate on target branch before public issue creation, if branch-scoped

Out of scope

• Do not implement PostgreSQL, S3-compatible storage, Valkey/Redis coordination, backup automation, cloud-provider scripts, Docker Compose, Kubernetes, Terraform, public /v1 authentication, backend decryption, or key custody changes in this documentation task.
• Do not include deployment-specific secrets, hostnames, credentials, private network details, or real incident data.

Notes

• Supported by docs/production-cluster-scope.md, docs/retention-backup-deletion.md, docs/deployment.md, docs/security-model.md, and docs/threat-model.md.
• Related PR: #88.
• Related open issue: #82 covers incident deletion and retention enforcement for current/future policy. This draft is separate because it is specific to production-cluster backend backup, restore, and failure handling.
• Label note: repository labels do not include operations or runbook; this draft uses docs, deployment, maintenance, and security.

[TheSilkky]

Closed in PR #112.