DigitalOcean Spaces S3 Backup Failing - "Can't Extract Backup ID" Error #536
Description
I am trying to set up daily backups for my PostgreSQL database using DigitalOcean Spaces (S3-compatible storage). However, the backup process fails with the following error:
{"level":"error","ts":"2025-03-17T12:00:28.666542951Z","logger":"barman","msg":"Can't extract backup id","backupName":"postgres-cluster-daily-backup-20250317115941","backupNamespace":"ambientiot","logging_pod":"postgres-cluster-1","command":"barman-cloud-backup-show","options":["--format","json","--endpoint-url","https://mybucket.fra1.digitaloceanspaces.com","--cloud-provider","aws-s3","s3://mybucket/ambientiot/postgres","postgres-cluster","backup-20250317120012"],"stdout":"","stderr":"2025-03-17 12:00:28,585 [663] ERROR: Barman cloud backup show exception: Unknown backup 'backup-20250317120012' for server 'postgres-cluster'\n","error":"exit status 4","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.0.0-20241219102532-2807bc88310d/pkg/log/log.go:125\ngithub.com/cloudnative-pg/barman-cloud/pkg/command.executeQueryCommand\n\tpkg/mod/github.com/cloudnative-pg/barman-cloud@v0.0.0-20241218093921-134c7de4954a/pkg/command/backuplist.go:88\ngithub.com/cloudnative-pg/barman-cloud/pkg/command.GetBackupByName\n\tpkg/mod/github.com/cloudnative-pg/barman-cloud@v0.0.0-20241218093921-134c7de4954a/pkg/command/backuplist.go:141\ngithub.com/cloudnative-pg/barman-cloud/pkg/backup.(*Command).GetExecutedBackupInfo\n\tpkg/mod/github.com/cloudnative-pg/barman-cloud@v0.0.0-20241218093921-134c7de4954a/pkg/backup/backup.go:153\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*BackupCommand).takeBackup\n\tpkg/management/postgres/backup.go:248\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*BackupCommand).run\n\tpkg/management/postgres/backup.go:174"}
{"level":"error","ts":"2025-03-17T12:00:28.666732883Z","msg":"Backup failed","backupName":"postgres-cluster-daily-backup-20250317115941","backupNamespace":"postgres-cluster-daily-backup-20250317115941","logging_pod":"postgres-cluster-1","error":"exit status 4","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.0.0-20241219102532-2807bc88310d/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*BackupCommand).run\n\tpkg/management/postgres/backup.go:178"}
The backup appears to be created, but it fails during the verification step, stating that the backup ID does not exist.
Helm Configuration:
type: postgis
version:
postgresql: "17"
postgis: "3.5"
cluster:
instances: 3
monitoring:
enabled: true
podMonitor:
enabled: true
affinity:
topologyKey: "kubernetes.io/hostname" # Spread across different nodes
storage:
size: 50Gi
resources:
requests:
memory: 8Gi
cpu: "2"
limits:
memory: 16Gi
cpu: "4"
postgresql:
parameters:
max_connections: "500"
shared_buffers: "4GB" # 25% of RAM
work_mem: "64MB"
maintenance_work_mem: "1GB"
effective_cache_size: "12GB"
# ---------------- AUTOVACUUM & ANALYZE ----------------
autovacuum: "on" # Keep it on to prevent table bloat
autovacuum_vacuum_cost_limit: "2000" # Increase vacuum efficiency
autovacuum_vacuum_scale_factor: "0.05" # Start vacuuming when 5% of table is changed
autovacuum_analyze_scale_factor: "0.02" # Analyze more frequently
backups:
enabled: true
endpointURL: https://mybucket.fra1.digitaloceanspaces.com
provider: s3
s3:
region: "fra1"
bucket: "mybucket"
path: "/ambientiot/postgres"
accessKey: "MY_ACCESS_KEY"
secretKey: "MY_SECRET"
scheduledBackups:
- name: daily-backup
schedule: "0 0 0 * * *" # Daily at midnight
backupOwnerReference: self
retentionPolicy: "30d"
poolers:
- name: rw
type: rw
instances: 1
monitoring:
enabled: true
podMonitor:
enabled: true
relabelings:
- targetLabel: type
replacement: rw
- name: ro
type: ro
instances: 1
monitoring:
enabled: true
podMonitor:
enabled: true
relabelings:
- targetLabel: type
replacement: ro
kubectl describe backup postgres-cluster-daily-backup-20250317115941 -n ambientiot
Name: postgres-cluster-daily-backup-20250317115941
Namespace: ambientiot
Labels: cnpg.io/cluster=postgres-cluster
cnpg.io/immediateBackup=true
cnpg.io/scheduled-backup=postgres-cluster-daily-backup
Annotations: <none>
API Version: postgresql.cnpg.io/v1
Kind: Backup
Metadata:
Creation Timestamp: 2025-03-17T11:59:41Z
Generation: 1
Owner References:
API Version: postgresql.cnpg.io/v1
Controller: true
Kind: ScheduledBackup
Name: postgres-cluster-daily-backup
UID: 166882a4-953d-48f2-b9cf-6500c1f2c882
Resource Version: 8076954
UID: cf95ee45-7eb7-44a9-9b1b-769a66213ab4
Spec:
Cluster:
Name: postgres-cluster
Method: barmanObjectStore
Status:
Backup Name: backup-20250317120012
Destination Path: s3://mybackup/ambientiot/postgres
Encryption: AES256
Endpoint URL: https://mybackup.fra1.digitaloceanspaces.com
Error: exit status 4
Instance ID:
Container ID: containerd://f851fc5abea577d45368dca9fe69cbc9da89df22eb401b73a54160817585f126
Pod Name: postgres-cluster-1
Method: barmanObjectStore
Phase: failed
s3Credentials:
Access Key Id:
Key: ACCESS_KEY_ID
Name: postgres-cluster-backup-s3-creds
Secret Access Key:
Key: ACCESS_SECRET_KEY
Name: postgres-cluster-backup-s3-creds
Server Name: postgres-cluster
Stopped At: 2025-03-17T12:00:28Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FindingPod 3m40s (x2 over 3m40s) cloudnative-pg-backup Couldn't find target pod postgres-cluster-1, will retry in 30 seconds
Normal Starting 3m10s (x3 over 3m40s) cloudnative-pg-backup Starting backup for cluster postgres-cluster
Normal Starting 3m9s local-webserver Backup started
Normal Completed 2m53s local-webserver Backup completed
Normal Failed 2m53s local-webserver Backup failed