Logical backup incompatible with the main Postgres pod anti-affinity rules

[IlyaSemenov]

If enable_pod_antiaffinity is set to true, pods created by the logical backup cronjob are also started with anti-affinity. That doesn't make sense, and prevents logical backups on clusters where number of Postgres pods equals to the total number of nodes in the cluster (e.g. 2 nodes and 2 Postgres pods).

Expected

Logical backup pods are always started without anti-affinity (or at least it's possible to remove anti-affinity for logical backup while keeping it for database cluster pods).

If enable_pod_antiaffinity is set to true, pods created by the logical backup cronjob can not be scheduled to the same nodes because of matching labels. That prevents logical backups on clusters where number of Postgres pods equals to the total number of nodes in the cluster (e.g. 2 nodes and 2 Postgres pods).

[Jan-M]

Good catch, this does not sound right. Quite the opposite on a quick thought, we might want to have them as close to the master as possible.

I suggest we remove the anti affinity by default. And then we add a config option to put affinity on the logical backup job.

Thoughts?

[IlyaSemenov]

Sounds good to me.

By the way - regarding "as close to the master as possible" - I guess it makes more sense to run the logical backup off the replica server, not master. That would ease the load on master, and reduce delays caused by locking tables. It's completely unrelated to this particular issue, of course.

[FxKu]

Hm strange, in the code we disable anti-affinity for logical backup pod template. But you see that it gets created?

[IlyaSemenov]

The logical backup pod is indeed created with valid affinity rules, I described the problem incorrectly.

What I think happens is that the main Postgres pod's existing anti-affinity rules prevent the logical backup pod from being scheduled (to the same node).

This is what kubectl describe prints for the logical backup pod:

  Warning  FailedScheduling  2m58s (x1447 over 19h)  default-scheduler  0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules.

Here's the logical backup pod yaml:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2019-06-20T17:35:01Z"
  generateName: logical-backup-foo-db-1561052100-
  labels:
    application: spilo
    cluster-name: foo-db
    controller-uid: bb4f76bb-9381-11e9-8f3a-5254005c0982
    job-name: logical-backup-foo-db-1561052100
    team: foo
  name: logical-backup-foo-db-1561052100-x9vdm
  namespace: foo
  ownerReferences:
  - apiVersion: batch/v1
    blockOwnerDeletion: true
    controller: true
    kind: Job
    name: logical-backup-foo-db-1561052100
    uid: bb4f76bb-9381-11e9-8f3a-5254005c0982
  resourceVersion: "7131816"
  selfLink: /api/v1/namespaces/foo/pods/logical-backup-foo-db-1561052100-x9vdm
  uid: bb553a0a-9381-11e9-8f3a-5254005c0982
spec:
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              application: spilo-logical-backup
              version: foo-db
          topologyKey: kubernetes.io/hostname
        weight: 1
  containers:
  - env:
    - name: SCOPE
      value: foo-db
    - name: LOGICAL_BACKUP_S3_BUCKET
      value: foo-postgres-backups
    - name: LOGICAL_BACKUP_S3_BUCKET_SCOPE_SUFFIX
      value: /e223d869-7a53-11e9-8f3a-5254005c0982
    - name: PG_VERSION
      value: "11"
    - name: PGPORT
      value: "5432"
    - name: PGUSER
      value: postgres
    - name: PGDATABASE
      value: postgres
    - name: PGSSLMODE
      value: require
    - name: PGPASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: postgres.foo-db.credentials.postgresql.acid.zalan.do
    image: registry.opensource.zalan.do/acid/logical-backup
    imagePullPolicy: IfNotPresent
    name: logical-backup
    ports:
    - containerPort: 8008
      protocol: TCP
    - containerPort: 5432
      protocol: TCP
    - containerPort: 8080
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi
    securityContext:
      privileged: false
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: zalando-postgres-operator-token-fddcm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: zalando-postgres-operator
  serviceAccountName: zalando-postgres-operator
  terminationGracePeriodSeconds: 300
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: zalando-postgres-operator-token-fddcm
    secret:
      defaultMode: 420
      secretName: zalando-postgres-operator-token-fddcm
status:
  conditions:
  - lastProbeTime: "2019-06-21T13:33:12Z"
    lastTransitionTime: "2019-06-20T17:35:01Z"
    message: '0/1 nodes are available: 1 node(s) didn''t match pod affinity/anti-affinity,
      1 node(s) didn''t satisfy existing pods anti-affinity rules.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable

and here's the Postgres pod:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 10.42.0.151/32
    status: '{"conn_url":"postgres://10.42.0.151:5432/postgres","api_url":"http://10.42.0.151:8008/patroni","state":"running","role":"master","xlog_location":10955522048,"timeline":8}'
  creationTimestamp: "2019-06-02T17:28:46Z"
  generateName: foo-db-
  labels:
    application: spilo
    cluster-name: foo-db
    controller-revision-hash: foo-db-5589479ff7
    spilo-role: master
    statefulset.kubernetes.io/pod-name: foo-db-0
    team: foo
  name: foo-db-0
  namespace: foo
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: foo-db
    uid: de3e8709-855b-11e9-8f3a-5254005c0982
  resourceVersion: "7099682"
  selfLink: /api/v1/namespaces/foo/pods/foo-db-0
  uid: e04aba1b-855b-11e9-8f3a-5254005c0982
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            application: spilo
            cluster-name: foo-db
            team: foo
        topologyKey: kubernetes.io/hostname
  containers:
  - env:
    - name: SCOPE
      value: foo-db
    - name: PGROOT
      value: /home/postgres/pgdata/pgroot
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: PGUSER_SUPERUSER
      value: postgres
    - name: KUBERNETES_SCOPE_LABEL
      value: cluster-name
    - name: KUBERNETES_ROLE_LABEL
      value: spilo-role
    - name: KUBERNETES_LABELS
      value: application=spilo
    - name: PGPASSWORD_SUPERUSER
      valueFrom:
        secretKeyRef:
          key: password
          name: postgres.foo-db.credentials.postgresql.acid.zalan.do
    - name: PGUSER_STANDBY
      value: standby
    - name: PGPASSWORD_STANDBY
      valueFrom:
        secretKeyRef:
          key: password
          name: standby.foo-db.credentials.postgresql.acid.zalan.do
    - name: PAM_OAUTH2
      value: https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees
    - name: HUMAN_ROLE
      value: zalandos
    - name: SPILO_CONFIGURATION
      value: '{"postgresql":{"bin_dir":"/usr/lib/postgresql/11/bin"},"bootstrap":{"initdb":[{"auth-host":"md5"},{"auth-local":"trust"}],"users":{"zalandos":{"password":"","options":["CREATEDB","NOLOGIN"]}},"dcs":{}}}'
    - name: WAL_S3_BUCKET
      value: foo-postgres-wal
    - name: WAL_BUCKET_SCOPE_SUFFIX
      value: /e223d869-7a53-11e9-8f3a-5254005c0982
    - name: WAL_BUCKET_SCOPE_PREFIX
    - name: DCS_ENABLE_KUBERNETES_API
      value: "true"
    - name: AWS_ACCESS_KEY_ID
      value: ...
    - name: AWS_SECRET_ACCESS_KEY
      value: ...
    - name: DEVELOP
      value: "1"
    - name: SPILO_PROVIDER
      value: local
    image: registry.opensource.zalan.do/acid/spilo-cdp-11:1.5-p76
    imagePullPolicy: IfNotPresent
    name: postgres
    ports:
    - containerPort: 8008
      protocol: TCP
    - containerPort: 5432
      protocol: TCP
    - containerPort: 8080
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi
    securityContext:
      privileged: false
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /home/postgres/pgdata
      name: pgdata
    - mountPath: /dev/shm
      name: dshm
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: zalando-postgres-operator-token-fddcm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: foo-db-0
  nodeName: kube-a0
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: zalando-postgres-operator
  serviceAccountName: zalando-postgres-operator
  subdomain: foo-db
  terminationGracePeriodSeconds: 300
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: pgdata
    persistentVolumeClaim:
      claimName: pgdata-foo-db-0
  - emptyDir:
      medium: Memory
    name: dshm
  - name: zalando-postgres-operator-token-fddcm
    secret:
      defaultMode: 420
      secretName: zalando-postgres-operator-token-fddcm
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-06-02T17:28:46Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-06-02T17:28:48Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-06-02T17:28:48Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-06-02T17:28:46Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://f4f2876aadf1958209184b6b05291a93b4b42d0a675330d8581c3ab58bc8c8a1
    image: registry.opensource.zalan.do/acid/spilo-cdp-11:1.5-p76
    imageID: docker-pullable://registry.opensource.zalan.do/acid/spilo-cdp-11@sha256:021ff1a29da0621d4aa2a4bd85c249cbe8b3c0c8a9b2826bd500c18175525565
    lastState: {}
    name: postgres
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2019-06-02T17:28:47Z"
  hostIP: 37.46.130.195
  phase: Running
  podIP: 10.42.0.151
  qosClass: Burstable
  startTime: "2019-06-02T17:28:46Z"

[IlyaSemenov]

Confirmed that by manually editing cronjob's manifest to:

spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
          labels:
            application: spilo1 <-----

the job pods started to run. (they wouldn't finish due to other errors though.)