Implement HP masternode backup strategy for testnet

[dashinfraclaw]

Status

A manual-only HP masternode backup flow has been prototyped in dash-network-deploy, scoped to hp_masternodes only.

Implemented so far:

• dedicated Ansible role to install backup tooling on HPMNs only
• separate install playbook and separate manual-run playbook
• manual GitHub Actions workflow scaffold for later UI-triggered runs
• kept isolated from normal deploy flow

Current backup behavior:

• no reboot
• no service stop
• no dashmate reset
• no container restart
• each HPMN creates its archive locally and uploads directly to S3

Storage

• bucket: s3://dash-testnet-hpmns-backups
• region: us-west-2
• public access blocked
• default server-side encryption enabled (AES256)
• object naming includes network, host, UTC timestamp, and trigger label

Prototype validation

Validated on hp-masternode-1:

• backup tooling installed successfully
• one manual backup run completed successfully
• upload to S3 verified
• object metadata and archive manifest verified

Test artifact:

• s3://dash-testnet-hpmns-backups/hpmn-backups/testnet/hp-masternode-1/20260414T111032Z_initial-test.tar.gz

Archive contents confirmed so far:

• dashmate config
• Tenderdash config directory
• gateway SSL files
• Core llmq directory

Next steps

• Confirm the real Tenderdash validator key/state paths and include them if they are outside the current archive set
• Perform a restore rehearsal on a disposable testnet HPMN and verify the backup is actually sufficient for recovery
• Document the restore procedure and recovery scope (identity/config recovery vs broader node recovery)
• Add retention/lifecycle policy for the S3 bucket
• If the rehearsal succeeds, roll out install + manual backup across all 29 HPMNs in batches rather than all at once

Related

• EBS snapshot audit and lifecycle policy #24 (EBS snapshot audit — old HP MN snapshots cleaned up)
• Create dash-infra Terraform project for shared services #21 (dash-infra Terraform — DLM policies should live here)