NSHM Backup Solution

AWS-native backup management CLI for NSHM datasets (ToshiAPI, THS, static reports, Weka). Replaces AWS Backup ($1,700 NZD/month before) with incremental S3 sync + 2-tier Glacier Instant Retrieval lifecycle and DynamoDB Point-in-Time exports — running at **$108 NZD/month** in production since May 2026 (per ADR-006).

Implementation Status

Phase	Status	Description
Phase 1	✅ Complete	CLI skeleton with Typer + Config system + S3 backup operations
Phase 2	✅ Complete	DynamoDB PITR export + EventBridge scheduling
Phase 3	✅ Complete	Slack + SNS-email; daily health report; Lambda-error alarm (ADR-005); YAML-managed recipients (ADR-008)
Phase 4	✅ Complete	Restore (S3 + DynamoDB, cross-account)
Phase 5	✅ Complete	Testing, validation, event audit log, daily-canary restore tests
Phase 6	✅ Complete	Production cutover (all 4 sources live since April 2026; AWS Backup decommissioned May 2026)
Phase 7	✅ Complete	ADR-009 signal-class taxonomy + class-1 head-check tagging; validated end-to-end against real AWS (sandbox cycles 0–3, June 2026)

Tests: 430+ passing · Coverage: 76% · Lint: ruff + mypy clean

Features (implemented)

Configuration: YAML config with Pydantic validation, alias→ARN mapping
S3 Backup: Incremental sync with 2-tier lifecycle policy — Standard → Glacier Instant Retrieval at 30 days; objects retained forever (ADR-006)
DynamoDB Backup: Point-in-Time export to S3, idempotent export bucket setup
EventBridge Scheduling: Create/enable/disable weekly/daily/hourly rules; localised time input and display
Lambda Handler: EventBridge-triggered backup orchestration (S3 + DynamoDB)
Restore: S3 (direct copy + S3 Batch Operations) and DynamoDB PITR restore with async status tracking
Testing: backup test integrity (ETag diff + PITR check) and backup test restore (sample restore)
Event audit log: Append-only JSONL log in backup bucket (_events/) — all backup/restore events recorded
Daily health report (backup health-report): per-source status + inventory freshness + source-vs-backup divergence (both directions) + class-1 head-check tagging + sampled restore verification, delivered to Slack and SNS-email. Fires automatically at 10:45 NZT (≈60 min after the 09:45 NZT scheduled backup); canary (weka) tested daily, large sources rotated through Mon/Wed/Fri. Signal classification follows ADR-009 (class 1 RED / class 2 ℹ info / class 3 yellow). See docs/user-guide/health-report.md.
Lambda-error alarm: CloudWatch alarm on backup Lambda Errors → SNS → email, fires within ~5 min of any hard failure. Complementary to the daily report (ADR-005 fast path).
YAML-managed notification recipients: notifications.alerts.emails + notifications.reports.email.addresses lists in backup-config.yaml; backup notifications apply reconciles SNS subscriptions to match. See docs/operations/enabling-notifications.md.
Dry-run mode: All mutating operations support --dry-run
JSON output: --output json for scripting
Localised timestamps: CLI input/output in NZDT/NZST/AEST/AEDT

Installation

uv sync --all-extras      # installs all deps including dev and docs extras

The backup command is registered as a console script and available immediately after install.

Usage

Global flags

backup --help
backup --dry-run run --source toshi     # Simulate without executing
backup --verbose run --source all       # Detailed logging
backup --output json schedule show      # Machine-readable output

Configuration

backup config show                      # Display full loaded config
backup config validate                  # Validate backup-config.yaml
backup config show --key retention      # Show a specific section

Run backup

backup run --source toshi               # S3 sync + DynamoDB PITR export for toshi
backup run --source ths                 # S3 sync for ths
backup run --source all                 # All sources
backup run --source toshi --full-sync   # Force full copy (skip ETag check)
backup --dry-run run --source toshi     # Preview without executing

Schedule management

backup schedule show                    # List EventBridge rules with localised run times

# --time accepts UTC (HH:MM), localised (HH:MM TZ), or full datetime (YYYY-MM-DD HH:MM TZ)
backup schedule add --source toshi --frequency weekly --time '02:00 NZST'
backup schedule add --source toshi --frequency weekly --time '2026-03-29 12:15 NZDT'  # day-of-week from date
backup schedule add --source toshi --frequency daily  --time '01:00 NZST'
backup schedule add --source toshi --frequency hourly --time '00:30'       # :30 past each hour

# Daily health report (ADR-005 slow path) — dispatched via the same Lambda
backup schedule add --source _health --task-type health_report \
    --frequency daily --time '14:30 NZST'

backup schedule enable --source toshi                      # Enable all rules for toshi
backup schedule enable --source toshi --frequency weekly   # Enable weekly only
backup schedule disable --source toshi                     # Disable all rules for toshi
backup schedule remove --source toshi --frequency daily    # Delete rule entirely

Restore

backup restore run --source toshi --buckets nzshm-toshi-api-data
backup restore run --source toshi --tables ToshiAPI-FileTable --to-point-in-time '2026-03-25 07:50 NZDT'
backup restore status --source toshi

Testing

backup test integrity --source toshi             # ETag diff + PITR check
backup test restore --source toshi               # Sample restore (direct copy)
backup test restore --source toshi --use-batch   # Sample restore via S3 Batch Operations
backup test alert                                # Force Lambda-error alarm to verify alarm path

Daily health report (ADR-005 slow path)

backup health-report preview                     # Build + print, skip restore tests
backup health-report run                         # Full report (incl. restore tests)
backup health-report run --send                  # Build + deliver via Slack + SNS-email
backup health-report run --weekday 0             # Force rotation (0=Mon … 6=Sun)

Operator guide: docs/user-guide/health-report.md.

Notification subscriptions

backup notifications show                        # List confirmed/pending subscribers
backup notifications apply                       # Reconcile SNS to match backup-config.yaml lists
backup notifications apply --dry-run             # Preview without changing SNS

Recipient runbook: docs/operations/enabling-notifications.md.

Event audit log

backup events --source toshi                  # Show recent backup/restore events
backup events --source toshi --limit 50

Status & reporting

backup status                                  # Live per-source backup state (implemented)
backup status --source toshi --output json     # Machine-readable status
backup events --source toshi                   # Recent backup/restore event log
backup report --period 30d                     # Aggregate report (stub — coming soon)
backup costs predict                           # Cost forecasting (stub — coming soon)

Configuration

Copy backup-config.example.yaml to backup-config.yaml and fill in your account ID and resource names:

general:
  region: ap-southeast-2
  environment: production
  lambda_arn: null          # Set after first serverless deploy

sources:
  toshi:
    display_name: "ToshiAPI"
    s3_buckets:
      - arn:aws:s3:::YOUR-TOSHI-BUCKET-NAME
    dynamodb_tables:
      - arn:aws:dynamodb:ap-southeast-2:ACCOUNT_ID:table/ToshiAPI-FileTable
      - arn:aws:dynamodb:ap-southeast-2:ACCOUNT_ID:table/ToshiAPI-ThingTable
    dynamodb_export_format: DYNAMODB_JSON

  ths:
    display_name: "THS_dataset_prod"
    s3_buckets:
      - arn:aws:s3:::YOUR-THS-BUCKET-NAME
    dynamodb_export_format: DYNAMODB_JSON

retention:
  hot_days: 30      # S3 Standard
  warm_days: 120    # Glacier Instant (must be >= hot_days + 90; Deep Archive transition derived from this)
  max_age_days: 365

restore:
  auto_approve_threshold: 100    # NZD — auto-approve below this
  dual_approval_threshold: 500   # NZD — two approvers above this

Deployment

Prerequisites

npm install -g serverless
uv sync --all-extras
cp backup-config.example.yaml backup-config.yaml   # edit with real values

Deploy Lambda

serverless deploy                  # Deploy to AWS
serverless deploy --stage prod     # Production stage

# After deploy, update lambda_arn in backup-config.yaml, then re-deploy
# to wire up EventBridge targets.

Add schedules after deploy

backup schedule add --source toshi --frequency weekly --time 14:00
backup schedule add --source ths   --frequency weekly --time 14:00
backup schedule show

Sandbox testing

See scripts/sandbox_setup.sh — creates lightweight source resources (S3 buckets + DynamoDB tables with PITR, seeded with sample data) in a sandbox AWS account so you can run backup run and backup schedule against real AWS without touching production.

# One-time setup
scripts/sandbox_setup.sh setup

# Run backup against sandbox resources
backup run --source toshi
backup run --source all --dry-run

# Tear down all sandbox resources when done
scripts/sandbox_setup.sh teardown

See backup-config.sandbox.yaml for the matching config.

Development

make test                 # All tests with coverage
make lint                 # ruff + mypy
make fmt                  # ruff format + ruff --fix
make check                # lint then test
make upgrade              # upgrade deps (1-week safety margin)

uv run pytest tests/test_foo.py   # single file

Architecture

EventBridge (cron)
  ├── per-source backup at 13:05 NZST → Lambda → S3 sync + DynamoDB PITR export
  └── daily health-report at 14:30 NZST → Lambda → Slack + SNS-email

CloudWatch alarm on Lambda Errors → SNS alerts topic → email subscribers

Backup bucket naming (in the backup account, region-suffixed):

S3: bb-{source}-s3-{label}-{region}-{source_account_id}
DynamoDB export: bb-{source}-dynamo-{region}-{source_account_id}

IAM: the Lambda role denies s3:DeleteObject and s3:DeleteBucket on real backup buckets (delete-protected; lifecycle expiration still fires). Scoped Allow for both on the bb-restore-test-* name pattern so the restore-test workflow can clean up its temp buckets. DynamoDB restores always go to a new table (never overwrite in-place).

Documentation

Operations:

Operator Cheatsheet — "to change X, do Y"
Daily Health Report (user guide)
Enabling Notifications (runbook)
Inventory Bucket Recovery (runbook)
Production Deployment Log — chronological deploy history with verification commands

Architecture / design:

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 346 Commits
.github/workflows		.github/workflows
docs		docs
scripts		scripts
src/aws_snapshot		src/aws_snapshot
tests		tests
.DS_Store		.DS_Store
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
backup-config.example.yaml		backup-config.example.yaml
mkdocs.yml		mkdocs.yml
pyproject.toml		pyproject.toml
sam.yaml		sam.yaml
samconfig.example.toml		samconfig.example.toml
setup.cfg		setup.cfg
uv.lock		uv.lock

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

NSHM Backup Solution

Implementation Status

Features (implemented)

Installation

Usage

Global flags

Configuration

Run backup

Schedule management

Restore

Testing

Daily health report (ADR-005 slow path)

Notification subscriptions

Event audit log

Status & reporting

Configuration

Deployment

Prerequisites

Deploy Lambda

Add schedules after deploy

Sandbox testing

Development

Architecture

Documentation

License

About

Uh oh!

Releases 1

Packages

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

NSHM Backup Solution

Implementation Status

Features (implemented)

Installation

Usage

Global flags

Configuration

Run backup

Schedule management

Restore

Testing

Daily health report (ADR-005 slow path)

Notification subscriptions

Event audit log

Status & reporting

Configuration

Deployment

Prerequisites

Deploy Lambda

Add schedules after deploy

Sandbox testing

Development

Architecture

Documentation

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 1

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages