feat(backup): configuration backup service (collect ingest + backend UI API)

[edospadoni]

Preview link

• https://my-proxy-qa-pr-81.onrender.com

Summary

Adds a first-class configuration-backup subsystem for NS8 and NethSecurity clients, replacing the legacy external backupd.nethesis.it. Clients stream a GPG-encrypted snapshot of their configuration to collect; users consume it (list / download / delete) from backend with the usual Logto JWT and RBAC.

Storage is bring-your-own S3: the code targets any S3-compatible bucket (DigitalOcean Spaces, AWS S3, Cloudflare R2, Backblaze B2, MinIO, Garage). MY never holds the plaintext — the ciphertext is produced client-side on the client.

Architecture

┌──────────────┐ GPG blob, Basic auth  ┌──────────────┐   PutObject    ┌──────────────┐
│    Client    │ ────────────────────► │   collect    │ ──────────────►│   S3 bucket  │
│ (NS8 / NSEC) │ POST /systems/backups │  (ingest)    │                └──────▲───────┘
└──────────────┘                       └──────────────┘                       │ presigned GET
                                                                              │
                                                                      ┌──────────────┐
                                                                      │   backend    │
                                                                      │  (UI reads)  │
                                                                      └──────────────┘

S3 object key layout — one prefix per system, recognisable at a glance in any bucket browser:

{org_id}/{system_key}/{backup_id}.{ext}

backup_id is a UUIDv7 (time-ordered); system_key is the stable user-facing NETH-… identifier the client authenticates with.

What changed

collect (ingest + client-facing read)

• POST /api/systems/backups — stream body to S3 with SHA-256 io.TeeReader, metadata reconciled via same-key CopyObject, inline retention enforcement under a Redis SET NX lock, per-system ingest rate limit.
• GET /api/systems/backups — list own backups (paginated; never truncated at the S3 1000-item response cap).
• GET /api/systems/backups/:id — download own backup (via presigned URL for external clients; cross-tenant isolation enforced by S3 prefix).
• Cross-service auth invalidation bus on Redis pub/sub (my:auth:invalidate): when backend regenerates a system secret, collect drops the in-memory + Redis auth cache within ~30 ms.
• Per-system rate limit (6/min, 429 + Retry-After) and per-system / per-org aggregate quotas.

backend (UI-facing read + GDPR)

• GET /api/systems/:id/backups — list with size, SHA-256, uploader IP, aggregate quota counters.
• GET /api/systems/:id/backups/:backup_id/download — issues a short-lived presigned URL (TTL capped server-side at 15 m).
• DELETE /api/systems/:id/backups/:backup_id — deletes one backup.
• purgeSystemBackups runs inline on DestroySystem for GDPR Article 17 (right to erasure); destroy is refused if the purge fails, so the operator can retry.
• Same RBAC as GET /systems/:id — a user sees a system's backups only if their org covers the system creator in the Owner → Distributor → Reseller → Customer hierarchy.

Storage abstraction

• Shared storage/s3.go in both services. Runs against any S3-compatible endpoint; backend and collect can use separate credentials scoped to read-only vs. write-only on the same bucket to contain blast radius if one service is ever compromised.

Docs

• New user-facing page under docs/systems/backups (EN + IT).
• collect/README.md + backend/README.md cover env setup, bucket layout, split credentials, and end-to-end round-trip commands.
• AGENTS.md updated with the new endpoint family and key layout.

Client API

Base URL: https://<collect-host> (whatever collect is deployed on — in prod my.nethesis.it).

All three endpoints use HTTP Basic auth with the system_key:system_secret pair returned at system registration.

Upload a backup

curl --user "$SYSTEM_KEY:$SYSTEM_SECRET" \
     -H "Content-Type: application/octet-stream" \
     -H "X-Filename: dump.json.gz.gpg" \
     --data-binary @/path/to/backup.gpg \
     https://<collect-host>/api/systems/backups

201 Created on success:

{
  "code": 201,
  "message": "backup stored",
  "data": {
    "id": "019db047-c5f7-766c-bae6-6f407da3e1c9.gpg",
    "filename": "dump.json.gz.gpg",
    "size": 130,
    "sha256": "b7d974f638ed5d07696dfce118fdbb20e2a447edfb3c79e563742925440c75b0",
    "mimetype": "application/octet-stream",
    "uploaded_at": "2026-04-21T13:43:07.502668Z",
    "uploader_ip": "127.0.0.1"
  }
}

Failure modes the client should handle: 401 (invalid/expired credentials → re-fetch from MY), 413 (body too large → trim or split), 429 Retry-After: N (rate-limited → back off N seconds), 5xx (transient → exponential retry).

List own backups

curl --user "$SYSTEM_KEY:$SYSTEM_SECRET" \
     https://<collect-host>/api/systems/backups

{
  "code": 200,
  "message": "backups listed",
  "data": {
    "backups": [
      {
        "id": "019db047-c5f7-766c-bae6-6f407da3e1c9.gpg",
        "filename": "dump.json.gz.gpg",
        "size": 130,
        "sha256": "b7d974f638ed5d07696dfce118fdbb20e2a447edfb3c79e563742925440c75b0",
        "uploaded_at": "2026-04-21T13:43:07.322Z",
        "uploader_ip": "127.0.0.1"
      }
    ]
  }
}

Pagination is handled server-side; the client receives the full list.

Download a backup

curl --user "$SYSTEM_KEY:$SYSTEM_SECRET" \
     -o restore.gpg \
     https://<collect-host>/api/systems/backups/019db047-c5f7-766c-bae6-6f407da3e1c9.gpg

Streams the ciphertext body with Content-Type: application/octet-stream. The client decrypts locally with its own GPG passphrase. Cross-tenant reads return 404 (a foreign backup_id simply does not exist under the caller's S3 prefix).

Client-side dedup (optional)

Before uploading, the client can compute the local MD5 of the just-encrypted blob and compare it with the sha256 of the previous run. If unchanged, the upload can be skipped to save bandwidth. Retention and quota enforcement live server-side regardless.

Deployment notes

New environment variables — identical on backend and collect:

BACKUP_S3_ENDPOINT=https://ams3.digitaloceanspaces.com
BACKUP_S3_REGION=ams3
BACKUP_S3_BUCKET=my-backups
BACKUP_S3_ACCESS_KEY=...
BACKUP_S3_SECRET_KEY=...
BACKUP_S3_USE_PATH_STYLE=false   # true only for local MinIO/Garage
#BACKUP_PRESIGN_TTL=5m            # capped at 15m server-side

Both .env.example and render.yaml are updated. Credentials can be split between backend (read-only) and collect (write-only) on providers that support per-key policies.

[github-actions]

🗑️ Redirect URIs Removed from Logto

The following redirect URIs have been automatically removed from the Logto application configuration:

Redirect URIs:

• https://my-proxy-qa-pr-81.onrender.com/login-redirect

Post-logout redirect URIs:

• https://my-proxy-qa-pr-81.onrender.com/login

Cleanup completed for PR #81.

[github-actions]

🔗 Redirect URIs Added to Logto

The following redirect URIs have been automatically added to the Logto application configuration:

Redirect URIs:

• https://my-proxy-qa-pr-81.onrender.com/login-redirect

Post-logout redirect URIs:

• https://my-proxy-qa-pr-81.onrender.com/login

These will be automatically removed when the PR is closed or merged.