How to setup standby cluster?

[hausanstreicha]

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.6.3
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
• Are you running Postgres Operator in production? no
• Type of issue? question

I'm having trouble setting up a standby cluster using ceph s3 as wal_path. I'm using this yaml for the standby cluster:

apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: acid-standby-cluster
  namespace: default
spec:
  numberOfInstances: 1
  postgresql:
    version: "13"
  standby:
    s3_wal_path: s3://s3bucket/spilo/acid-postgres-cluster/wal/13/
  teamId: acid
  volume:
    size: 1Gi
    storageClass: rook-ceph-block

I have created the following configmap with the relevant credentials and information. I have passed the configmap to the operator:

apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: key
  AWS_S3_FORCE_PATH_STYLE: "true"
  AWS_SECRET_ACCESS_KEY: secret
  BACKUP_NUM_TO_RETAIN: "14"
  BACKUP_SCHEDULE: 0 */12 * * *
  USE_WALG_BACKUP: "true"
  USE_WALG_RESTORE: "false"
  WAL_S3_BUCKET: s3bucket
  WALE_S3_ENDPOINT: s3endpoint
  WALG_DISABLE_S3_SSE: "false"
kind: ConfigMap
metadata:
  name: postgres-pod-config
  namespace: default

Logs look like this the whole time:

acid-standby-cluster-0 postgres 2021-08-26 11:34:04,809 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,814 - bootstrapping - INFO - Could not connect to 169.254.169.254, assuming local Docker setup
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,815 - bootstrapping - INFO - No meta-data available for this provider
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,815 - bootstrapping - INFO - Looks like your running local
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,839 - bootstrapping - INFO - Configuring bootstrap
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,839 - bootstrapping - INFO - Configuring pgqd
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,839 - bootstrapping - INFO - Configuring crontab
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,839 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,852 - bootstrapping - INFO - Configuring patroni
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,859 - bootstrapping - INFO - Writing to file /run/postgres.yml
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,860 - bootstrapping - INFO - Configuring pgbouncer
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,860 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,860 - bootstrapping - INFO - Configuring standby-cluster
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,860 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/WALE_S3_PREFIX
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,860 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/WALG_S3_PREFIX
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,861 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/AWS_REGION
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,861 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/AWS_INSTANCE_PROFILE
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,861 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/WALG_S3_SSE
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,861 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/USE_WALG_BACKUP
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,861 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/USE_WALG_RESTORE
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,861 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/WALE_LOG_DESTINATION
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,862 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-standby/TMPDIR
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,862 - bootstrapping - INFO - Configuring pam-oauth2
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,862 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,862 - bootstrapping - INFO - Configuring log
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,862 - bootstrapping - INFO - Configuring certificate
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,862 - bootstrapping - INFO - Generating ssl certificate
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,934 - bootstrapping - INFO - Configuring wal-e
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_S3_PREFIX
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_S3_PREFIX
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_ACCESS_KEY_ID
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_S3_ENDPOINT
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_ENDPOINT
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_DISABLE_S3_SSE
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,935 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_S3_FORCE_PATH_STYLE
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,936 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_DOWNLOAD_CONCURRENCY
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,936 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_UPLOAD_CONCURRENCY
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,936 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/USE_WALG_BACKUP
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,936 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_LOG_DESTINATION
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,936 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/BACKUP_NUM_TO_RETAIN
acid-standby-cluster-0 postgres 2021-08-26 11:34:06,936 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/TMPDIR
acid-standby-cluster-0 postgres 2021-08-26 11:34:08,196 INFO: Selected new K8s API server endpoint https://10.83.16.11:6443
acid-standby-cluster-0 postgres 2021-08-26 11:34:08,208 INFO: No PostgreSQL configuration items changed, nothing to reload.
acid-standby-cluster-0 postgres 2021-08-26 11:34:08,211 INFO: Lock owner: None; I am acid-standby-cluster-0
acid-standby-cluster-0 postgres 2021-08-26 11:34:08,298 INFO: trying to bootstrap (without leader)
acid-standby-cluster-0 postgres 2021-08-26 11:34:18,715 INFO: Lock owner: None; I am acid-standby-cluster-0
acid-standby-cluster-0 postgres 2021-08-26 11:34:18,715 INFO: not healthy enough for leader race
acid-standby-cluster-0 postgres 2021-08-26 11:34:18,768 INFO: bootstrap (without leader) in progress

I have setup local backups and WAL to ceph s3 on my other cluster and everything is working as intended. Everything is getting uploaded:

2021-08-25 13:12        13945  s3://s3bucket/spilo/acid-postgres-cluster/7ee78e65-aaf1-4889-986e-1e79df856b57/logical_backups/1629897149.sql.gz
2021-08-26 09:00          269  s3://s3bucket/spilo/acid-postgres-cluster/wal/13/wal_005/00000015000000000000003C.00000028.backup.lz4
2021-08-26 09:01      7495912  s3://s3bucket/spilo/acid-postgres-cluster/wal/13/basebackups_005/base_00000015000000000000003E/tar_partitions/part_001.tar.lz4
2021-08-26 09:01          536  s3://s3bucket/spilo/acid-postgres-cluster/wal/13/basebackups_005/base_00000015000000000000003E/tar_partitions/part_003.tar.lz4
2021-08-26 09:01          504  s3://s3bucket/spilo/acid-postgres-cluster/wal/13/basebackups_005/base_00000015000000000000003E/tar_partitions/pg_control.tar.lz4

Am I'm missing something or did I misconfigure anything?

[FxKu]

Have a look at #1150. I think, copying the set of env variables and prepending STANDBY_ will help here.

[hausanstreicha]

Thank you for your help. It's working now.