RFC: What backup and restore API should DBaaS expose for M1?

[Yohansenanayake]

What we are asking

We need to agree on the user-facing backup and restore contract for the DBaaS M1 release.

The proposal follows the core Amazon RDS experience while keeping M1 narrow:

• PostgreSQL only;
• tenant-owned S3-compatible backup storage;
• automated backups with point-in-time recovery (PITR);
• durable manual and final snapshots;
• restore to a new DB instance only.

This discussion is intentionally limited to API concepts and observable behavior. Backup tooling, VM orchestration, repository layout, retention execution, and other implementation details can be introduced in follow-up comments.

Proposed API concepts

API concept	Purpose
BackupRepository	One DB instance's isolated destination in tenant-owned object storage
DBInstance.spec.backup	Automated backup, PITR retention, backup window, and deletion policy
DBSnapshot	A named manual or final recovery point
DBRestore	A request to restore a snapshot or point in time into a new DB instance

1. BackupRepository

A repository is not the S3 bucket itself. It is one database's isolated destination: bucket, prefix, and credentials.

apiVersion: dbaas.opencloud.wso2.com/v1alpha1
kind: BackupRepository
metadata:
  name: orders-backups
  namespace: tenant-finance
spec:
  databaseRef:
    name: orders-db
  storage:
    existing:
      provider: s3
      s3:
        endpoint: https://s3.example.com
        bucket: finance-database-backups
        prefix: databases/orders-db
        region: us-east-1
        credentialsSecretRef:
          name: orders-backup-credentials

Proposed behavior:

• The repository and DB instance must be in the same namespace.
• One repository binds permanently to at most one concrete DB instance UID.
• The controller records the resolved UID in status.binding.databaseUID
• A deleted DB instance does not make its repository reusable by another DB instance with the same name.
• Multiple repositories may use one physical bucket when they use different prefixes and appropriately scoped credentials.
• The tenant owns the bucket, prefix permissions, credentials, and external storage policy.
• M1 supports existing S3-compatible storage. The API reserves a future managed-storage branch for COSI without requiring it now.

The repository is optional when the DB instance has no automated backups and no final-snapshot policy.

2. Automated backups and PITR

spec:
  backup:
    repositoryRef:
      name: orders-backups
    retentionPeriodDays: 7
    preferredWindow: "02:00-02:30"
    deletionPolicy:
      finalSnapshot: false

Proposed behavior:

• retentionPeriodDays: 0 disables automated backups and PITR.
• M1 supports a positive retention period from 1 through 35 days.
• Positive retention requires a ready repository bound to that DB instance.
• The preferred window is a UTC start window, not a completion deadline.
• The user configures retention and the backup window, not the effective PITR window.
• DBaaS reports the effective earliestRestorableTime and latestRestorableTime based on recovery data that is actually usable.
• Database readiness and backup/PITR readiness are reported independently.

Example status:

status:
  backup:
    automatedBackupsEnabled: true
    lastSuccessfulBackupTime: "2026-08-03T02:14:00Z"
    earliestRestorableTime: "2026-07-27T02:14:00Z"
    latestRestorableTime: "2026-08-03T10:55:00Z"
    conditions:
      - type: ContinuousRecoveryReady
        status: "True"
        reason: RecoveryWindowAvailable

3. Manual and final snapshots

A manual snapshot is a first-class, immutable DBSnapshot resource:

apiVersion: dbaas.opencloud.wso2.com/v1alpha1
kind: DBSnapshot
metadata:
  name: orders-before-upgrade
  namespace: tenant-finance
spec:
  dbInstanceRef:
    name: orders-db
  repositoryRef:
    name: orders-backups

Proposed behavior:

• A snapshot represents the whole managed PostgreSQL instance.
• Snapshot creation is asynchronous: Pending → Creating → Available or Failed.
• Manual snapshots do not expire automatically in M1.
• Manual snapshots survive deletion of the source DB instance.
• A DB created without a repository can later bind a repository and take a manual snapshot without enabling automated backups.
• Snapshot status records the source DB instance UID and both the repository name and UID.
• Restore validates the recorded repository UID, so recreating a repository with the same name cannot redirect a snapshot.

The DB instance deletion policy can request a final snapshot:

spec:
  backup:
    repositoryRef:
      name: orders-backups
    retentionPeriodDays: 0
    deletionPolicy:
      finalSnapshot: true
      finalSnapshotName: orders-db-final

The final snapshot must become Available before destructive deletion continues. It then follows the same durable lifecycle as a manual snapshot.

For M1, automated PITR data is removed during DB instance teardown. Retaining automated PITR after source deletion, as a separate restorable resource, is proposed as a later feature.

4. Restore

Restore always creates a new DB instance and leaves the source unchanged.

apiVersion: dbaas.opencloud.wso2.com/v1alpha1
kind: DBRestore
metadata:
  name: restore-orders
  namespace: tenant-finance
spec:
  source:
    snapshotRef:
      name: orders-before-upgrade
  target:
    dbInstanceName: orders-restored
    dbInstanceClass: db.m5.large
    allocatedStorage: 150
    networkRef: tenant-finance/database-network
    credentials:
      mode: generated
    backup:
      inheritPolicy: true
      repositoryRef:
        name: orders-restored-backups

PITR selects an active source DB instance and either a UTC timestamp inside its reported recovery window or useLatestRestorableTime: true.

Example restoring orders-db to a specific point in time:

apiVersion: dbaas.opencloud.wso2.com/v1alpha1
kind: DBRestore
metadata:
  name: restore-orders-to-time
  namespace: tenant-finance
spec:
  source:
    pointInTime:
      dbInstanceRef:
        name: orders-db
      restoreTime: "2026-08-03T10:42:30Z"
  target:
    dbInstanceName: orders-restored
    dbInstanceClass: db.m5.large
    allocatedStorage: 150
    networkRef: tenant-finance/database-network
    credentials:
      mode: generated
    backup:
      inheritPolicy: true
      repositoryRef:
        name: orders-restored-backups

The request is accepted only when restoreTime is between the source's reported earliestRestorableTime and latestRestorableTime. The source remains unchanged, while orders-restored is created with database state recovered to the requested UTC time.

Proposed restore behavior:

• Restore never overwrites or rewinds an existing DB instance.
• PostgreSQL data, tenant-created roles, grants, master username, and engine major version are recovered.
• The target gets a new name, endpoint, network identity, TLS identity, and internal credentials.
• Static IP configuration is never inherited automatically.
• The target master password is freshly generated by default, or the user supplies a new target BYO password Secret.
• Historical master, replication, and exporter passwords recovered with the data are replaced before the target becomes Available.
• The target can inherit the source backup retention and window, but it cannot inherit the source repository. An enabled target uses a new repository bound to the target DB name and later UID.
• M1 restores only to the same PostgreSQL major version.
• M1 does not perform application cutover.

BackupRepository requirement matrix

Automated backups	Final snapshot policy	BackupRepository required when provisioning?
Disabled	Disabled	No
Enabled	Disabled	Yes
Disabled	Enabled	Yes
Enabled	Enabled	Yes

A manual snapshot always requires a ready repository, but that repository can be introduced after DB provisioning.

M1 non-goals

• MySQL backup execution;
• in-place restore;
• cross-tenant snapshot sharing or copy;
• cross-region backup replication;
• retained automated backups after source deletion;
• table-, schema-, or single-database restore;
• restore across PostgreSQL major versions;
• managed bucket provisioning through COSI;
• Azure Blob or Google Cloud Storage repositories;
• automatic application cutover.

Clarifications requested

1. Resource model: Do we agree with separate DBSnapshot and DBRestore resources, or should restore be expressed as DBInstance.spec.restoreFrom?
2. Repository binding: Should BackupRepository.spec.databaseRef.name plus the immutable UID recorded in status be the authoritative one-database binding?
3. Stopped instances: Should stopped time pause automated-backup retention as RDS does, or should M1 use elapsed wall-clock retention?
4. Final snapshot failure: If the tenant repository is unavailable during deletion, should deletion remain blocked indefinitely, or should M1 expose an explicit force-delete-without-final-snapshot action?
5. Restore defaults: Are the proposed inheritance rules correct—same engine major and master username, selectable class/storage, explicit target network, fresh TLS and credentials, inherited retention/window, and a new target repository?

Once these user-facing decisions are agreed, follow-up comments can cover backup tooling, storage layout, garbage collection, operation state machines, VM integration, credentials, and validation strategy.

[Yohansenanayake]

Where should the backup agent run?

In both approaches, PostgreSQL base backups and continuous WAL archiving are sent to S3-compatible object storage. We currently plan to use pgBackRest, but Barman remains an alternative. I propose we use approach 2 initially because of its advantages + the simplicity in implementation for the initial milestone.

Approach 1: dedicated backup server per namespace

A backup server inside each tenant namespace manages backups for all DB VMs in that namespace. Data first moves from each DB VM to the backup server (shown as rsync in the attached diagram), and the backup server sends it to S3.

Advantages

• Centralized backup configuration, credentials, scheduling, monitoring, and upgrades per namespace.
• Offloads compression, encryption, checksumming, and S3 upload work from the DB VMs, depending on the selected transfer design.
• Keeps S3 credentials out of individual DB VMs.
• Provides one place to limit aggregate backup concurrency and object-storage traffic for the namespace.
• Can provide local staging or caching for retries and restores.

Disadvantages

• A namespace-wide single point of failure and a larger failure blast radius.
• Can become a throughput bottleneck or noisy-neighbor point as the number and size of databases increase.
• Consumes additional CPU, memory, storage, and network quota from the tenant namespace.
• Introduces another data-transfer hop between each DB VM and S3.
• Requires lifecycle management, capacity planning, recovery, and possibly HA for the backup server itself.
• A compromised backup server may expose credentials or backup access for every database in the namespace.
• Still requires a secure network path from the backup server to every DB VM.

Approach 2: backup agent inside each DB VM

Each DB VM runs its own backup agent and sends base backups and WAL directly to S3.

Advantages

• No shared namespace-wide backup-server failure or bottleneck.
• No additional backup-server compute or storage charged to the namespace.
• Direct DB-to-S3 data path with no intermediate copy.
• Fewer infrastructure components and failure handoffs.
• The agent has a local view of PostgreSQL settings, data, and WAL state.
• Backup failures are isolated to one DB instance.

Disadvantages

• Backup compression, checksumming, encryption, and upload consume DB VM CPU, memory, disk I/O, and network bandwidth.
• S3 credentials and protected backup configuration exist in every DB VM.
• Guest-tool rollout, compatibility, and configuration must be managed across the VM fleet.
• Simultaneous backups from many DB VMs can create aggregate network or S3 pressure unless scheduling is coordinated.
• A compromised DB VM can expose that instance's backup repository access.

pgBackRest or Barman?

pgBackRest fits the per-VM approach well: it supports PostgreSQL base backups, continuous WAL archiving, full/differential/incremental backups, parallelism, retention, and direct S3-compatible repositories.

Barman is also worth considering. Its traditional centralized backup-server model aligns naturally with Approach 1, while its cloud utilities can archive WAL and backups to object storage. The choice should consider operational model, object-storage support, restore/PITR workflow, HA requirements, observability, performance, and team experience—not only backup creation.

Controller-to-guest command path

Challenge

The controller runs on the management network, while DB VMs run on isolated VM networks. There is no route the controller can use to configure or inspect a backup agent inside the guest. Attaching every DB VM to both the management and tenant VM networks would weaken network isolation and expand the attack surface.

Proposed path: KubeVirt serial console

Use the existing KubeVirt serial-console subresource as the control path:

Controller
  → Kubernetes API server
  → virt-api
  → virt-handler on the VMI's worker node
  → virt-launcher's serial Unix socket
  → restricted command wrapper inside the DB VM

The controller starts an HTTP WebSocket-upgrade request to the KubeVirt console subresource, resulting in a secure WebSocket stream. The Kubernetes API server routes the upgraded request to virt-api. virt-api opens a separate mutually-authenticated secure WebSocket connection to virt-handler on the node running the VMI. virt-handler bridges that stream to the VM's serial
console through the virt-serial0 Unix domain socket exposed by virt-launcher.

This reuses KubeVirt's existing control-plane path and does not require a route from the management network into the tenant VM network. We should not expose the unrestricted serial console to the controller. The guest should provide a restricted wrapper that accepts only a small, versioned allowlist of predefined backup operations. The wrapper design, protocol, authentication, validation, and privilege boundary will be described in a separate follow-up comment.

Feedback requested

1. Which execution model should we use for M1: one backup server per namespace or one backup agent per DB VM? What operational or failure scenarios are we missing?
2. If we choose a centralized backup server, what HA, capacity, credential isolation, and DB-to-backup-server transfer model should we require?
3. Is there a safer or simpler controller-to-guest path that preserves network isolation ?
4. Are there concerns with using the KubeVirt serial-console path for bounded management operations?
5. For this architecture, do you prefer pgBackRest or Barman, and why?