Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubernetes.core.k8s_scale doesn't scale statefulsets #503

Closed
haroonb opened this issue Aug 23, 2022 · 1 comment · Fixed by #579
Closed

kubernetes.core.k8s_scale doesn't scale statefulsets #503

haroonb opened this issue Aug 23, 2022 · 1 comment · Fixed by #579
Labels
jira type/bug Something isn't working verified The issue is reproduced

Comments

@haroonb
Copy link

haroonb commented Aug 23, 2022

SUMMARY
ISSUE TYPE
  • Bug Report
COMPONENT NAME

kubernetes.core.k8s_scale

ANSIBLE VERSION
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/butt/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 2.10.1
  libyaml = True
COLLECTION VERSION
 /usr/share/ansible/collections/ansible_collections
Collection      Version
--------------- -------
kubernetes.core 2.3.2

# /usr/lib/python3/dist-packages/ansible_collections
Collection      Version
--------------- -------
kubernetes.core 2.3.1
CONFIGURATION
[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core in version 2.15. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
CALLBACKS_ENABLED(/etc/ansible/ansible.cfg) = ['profile_tasks']
DEFAULT_JINJA2_NATIVE(/etc/ansible/ansible.cfg) = True
DEFAULT_LOG_PATH(/etc/ansible/ansible.cfg) = /var/log/ansible.log
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 300
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
OS / ENVIRONMENT

ubuntu 22.04

STEPS TO REPRODUCE
# this works
---
- hosts: opsia-m1
  gather_facts: no
  tasks:
  - name: service
    kubernetes.core.k8s:
      definition:
        apiVersion: v1
        kind: Service
        metadata:
          name: nginx
          namespace: default
          labels:
            app: nginx
        spec:
          ports:
          - port: 80
            name: web
          clusterIP: None
          selector:
            app: nginx

  - name: statefulset
    kubernetes.core.k8s:
      definition:
        apiVersion: apps/v1
        kind: StatefulSet
        metadata:
          name: web
          namespace: default
        spec:
          serviceName: "nginx"
          replicas: 3
          selector:
            matchLabels:
              app: nginx
          template:
            metadata:
              labels:
                app: nginx
            spec:
              containers:
              - name: nginx
                image: k8s.gcr.io/nginx-slim:0.8
                ports:
                - containerPort: 80
                  name: web
                volumeMounts:
                - name: www
                  mountPath: /usr/share/nginx/html
          volumeClaimTemplates:
          - metadata:
              name: www
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: rook-ceph-block
              resources:
                requests:
                  storage: 1Gi

  - name: scale down
    kubernetes.core.k8s_scale:
      api_version: v1
      kind: StatefulSet
      name: web
      namespace: default
      replicas: 0
      wait_timeout: 60

  - name: scale up
    kubernetes.core.k8s_scale:
      api_version: v1
      kind: StatefulSet
      name: web
      namespace: default
      replicas: 3
      wait_timeout: 60
# This doesn't work
---
- hosts: perftest-m1
  gather_facts: no
  tasks:

    - name: install
      command: kubectl apply -k github.com/zalando/postgres-operator/manifests

    - name: postgres
      kubernetes.core.k8s:
        definition: "{{ lookup('url', 'https://raw.githubusercontent.com/zalando/postgres-operator/v1.6.3/manifests/minimal-postgres-manifest.yaml', split_lines=False) }}"
        wait: yes
        wait_timeout: 60

    - name: check for statefulset
      ansible.builtin.wait_for:
        timeout: 60

    - name: scale down
      kubernetes.core.k8s_scale:
        api_version: v1
        kind: StatefulSet
        name: acid-minimal-cluster
        namespace: default
        replicas: 0
        wait_timeout: 60

    - name: scale up
      kubernetes.core.k8s_scale:
        api_version: v1
        kind: StatefulSet
        name: acid-minimal-cluster
        namespace: default
        replicas: 2
        wait_timeout: 60
EXPECTED RESULTS

The statefulset should be scaled down and up

ACTUAL RESULTS

The generated postgres statefulset is neither scaled down or up successfully. It always runs into the timout and fails. I have pasted the output below, when the statefulset is scaled up and the module tries to scale it down and vice versa.

This is my original issue to this #203

TASK [scale down] *************************
task path: 
Tuesday 23 August 2022  12:01:26 +0000 (0:01:00.463)       0:01:04.807 ********
    "changed": false,
    "duration": 60,
    "invocation": {
        "module_args": {
            "api_key": null,
            "api_version": "v1",
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "context": null,
            "continue_on_error": false,
            "current_replicas": null,
            "host": null,
            "impersonate_groups": null,
            "impersonate_user": null,
            "kind": "StatefulSet",
            "kubeconfig": null,
            "label_selectors": [],
            "name": "acid-minimal-cluster",
            "namespace": "default",
            "no_proxy": null,
            "password": null,
            "persist_config": null,
            "proxy": null,
            "proxy_headers": null,
            "replicas": 0,
            "resource_definition": null,
            "resource_version": null,
            "src": null,
            "username": null,
            "validate_certs": null,
            "wait": true,
            "wait_sleep": 5,
            "wait_timeout": 60
        }
    },
    "msg": "Resource scaling timed out",
    "result": {
        "apiVersion": "apps/v1",
        "kind": "StatefulSet",
        "metadata": {
            "creationTimestamp": "2022-08-23T11:46:00Z",
            "generation": 4,
            "labels": {
                "application": "spilo",
                "cluster-name": "acid-minimal-cluster",
                "team": "acid"
            },
            "managedFields": [
                {
                    "apiVersion": "apps/v1",
                    "fieldsType": "FieldsV1",
                    "fieldsV1": {
                        "f:metadata": {
                            "f:labels": {
                                ".": {},
                                "f:application": {},
                                "f:cluster-name": {},
                                "f:team": {}
                            }
                        },
                        "f:spec": {
                            "f:podManagementPolicy": {},
                            "f:revisionHistoryLimit": {},
                            "f:selector": {},
                            "f:serviceName": {},
                            "f:template": {
                                "f:metadata": {
                                    "f:labels": {
                                        ".": {},
                                        "f:application": {},
                                        "f:cluster-name": {},
                                        "f:team": {}
                                    },
                                    "f:namespace": {}
                                },
                                "f:spec": {
                                    "f:containers": {
                                        "k:{\"name\":\"postgres\"}": {
                                            ".": {},
                                            "f:env": {
                                                ".": {},
                                                "k:{\"name\":\"DCS_ENABLE_KUBERNETES_API\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"ENABLE_WAL_PATH_COMPAT\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"HUMAN_ROLE\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"KUBERNETES_LABELS\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"KUBERNETES_ROLE_LABEL\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"KUBERNETES_SCOPE_LABEL\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PAM_OAUTH2\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGPASSWORD_STANDBY\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:secretKeyRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"PGPASSWORD_SUPERUSER\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:secretKeyRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"PGROOT\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGUSER_STANDBY\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGUSER_SUPERUSER\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGVERSION\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"POD_IP\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:fieldRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"POD_NAMESPACE\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:fieldRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"SCOPE\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"SPILO_CONFIGURATION\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                }
                                            },
                                            "f:image": {},
                                            "f:imagePullPolicy": {},
                                            "f:name": {},
                                            "f:ports": {
                                                ".": {},
                                                "k:{\"containerPort\":5432,\"protocol\":\"TCP\"}": {
                                                    ".": {},
                                                    "f:containerPort": {},
                                                    "f:protocol": {}
                                                },
                                                "k:{\"containerPort\":8008,\"protocol\":\"TCP\"}": {
                                                    ".": {},
                                                    "f:containerPort": {},
                                                    "f:protocol": {}
                                                },
                                                "k:{\"containerPort\":8080,\"protocol\":\"TCP\"}": {
                                                    ".": {},
                                                    "f:containerPort": {},
                                                    "f:protocol": {}
                                                }
                                            },
                                            "f:readinessProbe": {
                                                ".": {},
                                                "f:failureThreshold": {},
                                                "f:httpGet": {
                                                    ".": {},
                                                    "f:path": {},
                                                    "f:port": {},
                                                    "f:scheme": {}
                                                },
                                                "f:initialDelaySeconds": {},
                                                "f:periodSeconds": {},
                                                "f:successThreshold": {},
                                                "f:timeoutSeconds": {}
                                            },
                                            "f:resources": {
                                                ".": {},
                                                "f:limits": {
                                                    ".": {},
                                                    "f:cpu": {},
                                                    "f:memory": {}
                                                },
                                                "f:requests": {
                                                    ".": {},
                                                    "f:cpu": {},
                                                    "f:memory": {}
                                                }
                                            },
                                            "f:securityContext": {
                                                ".": {},
                                                "f:allowPrivilegeEscalation": {},
                                                "f:privileged": {},
                                                "f:readOnlyRootFilesystem": {}
                                            },
                                            "f:terminationMessagePath": {},
                                            "f:terminationMessagePolicy": {},
                                            "f:volumeMounts": {
                                                ".": {},
                                                "k:{\"mountPath\":\"/dev/shm\"}": {
                                                    ".": {},
                                                    "f:mountPath": {},
                                                    "f:name": {}
                                                },
                                                "k:{\"mountPath\":\"/home/postgres/pgdata\"}": {
                                                    ".": {},
                                                    "f:mountPath": {},
                                                    "f:name": {}
                                                }
                                            }
                                        }
                                    },
                                    "f:dnsPolicy": {},
                                    "f:restartPolicy": {},
                                    "f:schedulerName": {},
                                    "f:securityContext": {},
                                    "f:serviceAccount": {},
                                    "f:serviceAccountName": {},
                                    "f:terminationGracePeriodSeconds": {},
                                    "f:volumes": {
                                        ".": {},
                                        "k:{\"name\":\"dshm\"}": {
                                            ".": {},
                                            "f:emptyDir": {
                                                ".": {},
                                                "f:medium": {}
                                            },
                                            "f:name": {}
                                        }
                                    }
                                }
                            },
                            "f:updateStrategy": {
                                "f:type": {}
                            },
                            "f:volumeClaimTemplates": {}
                        }
                    },
                    "manager": "postgres-operator",
                    "operation": "Update",
                    "time": "2022-08-23T11:46:00Z"
                },
                {
                    "apiVersion": "apps/v1",
                    "fieldsType": "FieldsV1",
                    "fieldsV1": {
                        "f:status": {
                            "f:availableReplicas": {},
                            "f:collisionCount": {},
                            "f:currentRevision": {},
                            "f:observedGeneration": {},
                            "f:replicas": {},
                            "f:updateRevision": {}
                        }
                    },
                    "manager": "kube-controller-manager",
                    "operation": "Update",
                    "subresource": "status",
                    "time": "2022-08-23T12:01:47Z"
                }
            ],
            "name": "acid-minimal-cluster",
            "namespace": "default",
            "resourceVersion": "18556721",
            "uid": "e3bdef61-7be8-4de6-8122-84f2192b6a2e"
        },
        "spec": {
            "podManagementPolicy": "OrderedReady",
            "replicas": 0,
            "revisionHistoryLimit": 10,
            "selector": {
                "matchLabels": {
                    "application": "spilo",
                    "cluster-name": "acid-minimal-cluster"
                }
            },
            "serviceName": "acid-minimal-cluster",
            "template": {
                "metadata": {
                    "creationTimestamp": null,
                    "labels": {
                        "application": "spilo",
                        "cluster-name": "acid-minimal-cluster",
                        "team": "acid"
                    },
                    "namespace": "default"
                },
                "spec": {
                    "containers": [
                        {
                            "env": [
                                {
                                    "name": "SCOPE",
                                    "value": "acid-minimal-cluster"
                                },
                                {
                                    "name": "PGROOT",
                                    "value": "/home/postgres/pgdata/pgroot"
                                },
                                {
                                    "name": "POD_IP",
                                    "valueFrom": {
                                        "fieldRef": {
                                            "apiVersion": "v1",
                                            "fieldPath": "status.podIP"
                                        }
                                    }
                                },
                                {
                                    "name": "POD_NAMESPACE",
                                    "valueFrom": {
                                        "fieldRef": {
                                            "apiVersion": "v1",
                                            "fieldPath": "metadata.namespace"
                                        }
                                    }
                                },
                                {
                                    "name": "PGUSER_SUPERUSER",
                                    "value": "postgres"
                                },
                                {
                                    "name": "KUBERNETES_SCOPE_LABEL",
                                    "value": "cluster-name"
                                },
                                {
                                    "name": "KUBERNETES_ROLE_LABEL",
                                    "value": "spilo-role"
                                },
                                {
                                    "name": "PGPASSWORD_SUPERUSER",
                                    "valueFrom": {
                                        "secretKeyRef": {
                                            "key": "password",
                                            "name": "postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do"
                                        }
                                    }
                                },
                                {
                                    "name": "PGUSER_STANDBY",
                                    "value": "standby"
                                },
                                {
                                    "name": "PGPASSWORD_STANDBY",
                                    "valueFrom": {
                                        "secretKeyRef": {
                                            "key": "password",
                                            "name": "standby.acid-minimal-cluster.credentials.postgresql.acid.zalan.do"
                                        }
                                    }
                                },
                                {
                                    "name": "PAM_OAUTH2",
                                    "value": "https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees"
                                },
                                {
                                    "name": "HUMAN_ROLE",
                                    "value": "zalandos"
                                },
                                {
                                    "name": "ENABLE_WAL_PATH_COMPAT",
                                    "value": "true"
                                },
                                {
                                    "name": "PGVERSION",
                                    "value": "13"
                                },
                                {
                                    "name": "KUBERNETES_LABELS",
                                    "value": "{\"application\":\"spilo\"}"
                                },
                                {
                                    "name": "SPILO_CONFIGURATION",
                                    "value": "{\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}}"
                                },
                                {
                                    "name": "DCS_ENABLE_KUBERNETES_API",
                                    "value": "true"
                                }
                            ],
                            "image": "registry.opensource.zalan.do/acid/spilo-14:2.1-p6",
                            "imagePullPolicy": "IfNotPresent",
                            "name": "postgres",
                            "ports": [
                                {
                                    "containerPort": 8008,
                                    "protocol": "TCP"
                                },
                                {
                                    "containerPort": 5432,
                                    "protocol": "TCP"
                                },
                                {
                                    "containerPort": 8080,
                                    "protocol": "TCP"
                                }
                            ],
                            "readinessProbe": {
                                "failureThreshold": 3,
                                "httpGet": {
                                    "path": "/readiness",
                                    "port": 8008,
                                    "scheme": "HTTP"
                                },
                                "initialDelaySeconds": 6,
                                "periodSeconds": 10,
                                "successThreshold": 1,
                                "timeoutSeconds": 5
                            },
                            "resources": {
                                "limits": {
                                    "cpu": "1",
                                    "memory": "500Mi"
                                },
                                "requests": {
                                    "cpu": "100m",
                                    "memory": "100Mi"
                                }
                            },
                            "securityContext": {
                                "allowPrivilegeEscalation": true,
                                "privileged": false,
                                "readOnlyRootFilesystem": false
                            },
                            "terminationMessagePath": "/dev/termination-log",
                            "terminationMessagePolicy": "File",
                            "volumeMounts": [
                                {
                                    "mountPath": "/home/postgres/pgdata",
                                    "name": "pgdata"
                                },
                                {
                                    "mountPath": "/dev/shm",
                                    "name": "dshm"
                                }
                            ]
                        }
                    ],
                    "dnsPolicy": "ClusterFirst",
                    "restartPolicy": "Always",
                    "schedulerName": "default-scheduler",
                    "securityContext": {},
                    "serviceAccount": "postgres-pod",
                    "serviceAccountName": "postgres-pod",
                    "terminationGracePeriodSeconds": 300,
                    "volumes": [
                        {
                            "emptyDir": {
                                "medium": "Memory"
                            },
                            "name": "dshm"
                        }
                    ]
                }
            },
            "updateStrategy": {
                "type": "OnDelete"
            },
            "volumeClaimTemplates": [
                {
                    "apiVersion": "v1",
                    "kind": "PersistentVolumeClaim",
                    "metadata": {
                        "creationTimestamp": null,
                        "labels": {
                            "application": "spilo",
                            "cluster-name": "acid-minimal-cluster",
                            "team": "acid"
                        },
                        "name": "pgdata"
                    },
                    "spec": {
                        "accessModes": [
                            "ReadWriteOnce"
                        ],
                        "resources": {
                            "requests": {
                                "storage": "1Gi"
                            }
                        },
                        "volumeMode": "Filesystem"
                    },
                    "status": {
                        "phase": "Pending"
                    }
                }
            ]
        },
        "status": {
            "availableReplicas": 0,
            "collisionCount": 0,
            "currentRevision": "acid-minimal-cluster-6f87b7d65d",
            "observedGeneration": 4,
            "replicas": 0,
            "updateRevision": "acid-minimal-cluster-6f87b7d65d"
        }
    }
}

TASK [scale up] ****************************************
Tuesday 23 August 2022  11:53:54 +0000 (0:00:00.903)       0:01:03.972 ********
fatal: [perftest-m1]: FAILED! => {
    "changed": false,
    "duration": 60,
    "invocation": {
        "module_args": {
            "api_key": null,
            "api_version": "v1",
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "context": null,
            "continue_on_error": false,
            "current_replicas": null,
            "host": null,
            "impersonate_groups": null,
            "impersonate_user": null,
            "kind": "StatefulSet",
            "kubeconfig": null,
            "label_selectors": [],
            "name": "acid-minimal-cluster",
            "namespace": "default",
            "no_proxy": null,
            "password": null,
            "persist_config": null,
            "proxy": null,
            "proxy_headers": null,
            "replicas": 2,
            "resource_definition": null,
            "resource_version": null,
            "src": null,
            "username": null,
            "validate_certs": null,
            "wait": true,
            "wait_sleep": 5,
            "wait_timeout": 60
        }
    },
    "msg": "Resource scaling timed out",
    "result": {
        "apiVersion": "apps/v1",
        "kind": "StatefulSet",
        "metadata": {
            "creationTimestamp": "2022-08-23T11:46:00Z",
            "generation": 3,
            "labels": {
                "application": "spilo",
                "cluster-name": "acid-minimal-cluster",
                "team": "acid"
            },
            "managedFields": [
                {
                    "apiVersion": "apps/v1",
                    "fieldsType": "FieldsV1",
                    "fieldsV1": {
                        "f:metadata": {
                            "f:labels": {
                                ".": {},
                                "f:application": {},
                                "f:cluster-name": {},
                                "f:team": {}
                            }
                        },
                        "f:spec": {
                            "f:podManagementPolicy": {},
                            "f:revisionHistoryLimit": {},
                            "f:selector": {},
                            "f:serviceName": {},
                            "f:template": {
                                "f:metadata": {
                                    "f:labels": {
                                        ".": {},
                                        "f:application": {},
                                        "f:cluster-name": {},
                                        "f:team": {}
                                    },
                                    "f:namespace": {}
                                },
                                "f:spec": {
                                    "f:containers": {
                                        "k:{\"name\":\"postgres\"}": {
                                            ".": {},
                                            "f:env": {
                                                ".": {},
                                                "k:{\"name\":\"DCS_ENABLE_KUBERNETES_API\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"ENABLE_WAL_PATH_COMPAT\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"HUMAN_ROLE\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"KUBERNETES_LABELS\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"KUBERNETES_ROLE_LABEL\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"KUBERNETES_SCOPE_LABEL\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PAM_OAUTH2\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGPASSWORD_STANDBY\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:secretKeyRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"PGPASSWORD_SUPERUSER\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:secretKeyRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"PGROOT\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGUSER_STANDBY\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGUSER_SUPERUSER\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"PGVERSION\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"POD_IP\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:fieldRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"POD_NAMESPACE\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:valueFrom": {
                                                        ".": {},
                                                        "f:fieldRef": {}
                                                    }
                                                },
                                                "k:{\"name\":\"SCOPE\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                },
                                                "k:{\"name\":\"SPILO_CONFIGURATION\"}": {
                                                    ".": {},
                                                    "f:name": {},
                                                    "f:value": {}
                                                }
                                            },
                                            "f:image": {},
                                            "f:imagePullPolicy": {},
                                            "f:name": {},
                                            "f:ports": {
                                                ".": {},
                                                "k:{\"containerPort\":5432,\"protocol\":\"TCP\"}": {
                                                    ".": {},
                                                    "f:containerPort": {},
                                                    "f:protocol": {}
                                                },
                                                "k:{\"containerPort\":8008,\"protocol\":\"TCP\"}": {
                                                    ".": {},
                                                    "f:containerPort": {},
                                                    "f:protocol": {}
                                                },
                                                "k:{\"containerPort\":8080,\"protocol\":\"TCP\"}": {
                                                    ".": {},
                                                    "f:containerPort": {},
                                                    "f:protocol": {}
                                                }
                                            },
                                            "f:readinessProbe": {
                                                ".": {},
                                                "f:failureThreshold": {},
                                                "f:httpGet": {
                                                    ".": {},
                                                    "f:path": {},
                                                    "f:port": {},
                                                    "f:scheme": {}
                                                },
                                                "f:initialDelaySeconds": {},
                                                "f:periodSeconds": {},
                                                "f:successThreshold": {},
                                                "f:timeoutSeconds": {}
                                            },
                                            "f:resources": {
                                                ".": {},
                                                "f:limits": {
                                                    ".": {},
                                                    "f:cpu": {},
                                                    "f:memory": {}
                                                },
                                                "f:requests": {
                                                    ".": {},
                                                    "f:cpu": {},
                                                    "f:memory": {}
                                                }
                                            },
                                            "f:securityContext": {
                                                ".": {},
                                                "f:allowPrivilegeEscalation": {},
                                                "f:privileged": {},
                                                "f:readOnlyRootFilesystem": {}
                                            },
                                            "f:terminationMessagePath": {},
                                            "f:terminationMessagePolicy": {},
                                            "f:volumeMounts": {
                                                ".": {},
                                                "k:{\"mountPath\":\"/dev/shm\"}": {
                                                    ".": {},
                                                    "f:mountPath": {},
                                                    "f:name": {}
                                                },
                                                "k:{\"mountPath\":\"/home/postgres/pgdata\"}": {
                                                    ".": {},
                                                    "f:mountPath": {},
                                                    "f:name": {}
                                                }
                                            }
                                        }
                                    },
                                    "f:dnsPolicy": {},
                                    "f:restartPolicy": {},
                                    "f:schedulerName": {},
                                    "f:securityContext": {},
                                    "f:serviceAccount": {},
                                    "f:serviceAccountName": {},
                                    "f:terminationGracePeriodSeconds": {},
                                    "f:volumes": {
                                        ".": {},
                                        "k:{\"name\":\"dshm\"}": {
                                            ".": {},
                                            "f:emptyDir": {
                                                ".": {},
                                                "f:medium": {}
                                            },
                                            "f:name": {}
                                        }
                                    }
                                }
                            },
                            "f:updateStrategy": {
                                "f:type": {}
                            },
                            "f:volumeClaimTemplates": {}
                        }
                    },
                    "manager": "postgres-operator",
                    "operation": "Update",
                    "time": "2022-08-23T11:46:00Z"
                },
                {
                    "apiVersion": "apps/v1",
                    "fieldsType": "FieldsV1",
                    "fieldsV1": {
                        "f:status": {
                            "f:availableReplicas": {},
                            "f:collisionCount": {},
                            "f:currentReplicas": {},
                            "f:currentRevision": {},
                            "f:observedGeneration": {},
                            "f:readyReplicas": {},
                            "f:replicas": {},
                            "f:updateRevision": {},
                            "f:updatedReplicas": {}
                        }
                    },
                    "manager": "kube-controller-manager",
                    "operation": "Update",
                    "subresource": "status",
                    "time": "2022-08-23T11:54:12Z"
                }
            ],
            "name": "acid-minimal-cluster",
            "namespace": "default",
            "resourceVersion": "18554564",
            "uid": "e3bdef61-7be8-4de6-8122-84f2192b6a2e"
        },
        "spec": {
            "podManagementPolicy": "OrderedReady",
            "replicas": 2,
            "revisionHistoryLimit": 10,
            "selector": {
                "matchLabels": {
                    "application": "spilo",
                    "cluster-name": "acid-minimal-cluster"
                }
            },
            "serviceName": "acid-minimal-cluster",
            "template": {
                "metadata": {
                    "creationTimestamp": null,
                    "labels": {
                        "application": "spilo",
                        "cluster-name": "acid-minimal-cluster",
                        "team": "acid"
                    },
                    "namespace": "default"
                },
                "spec": {
                    "containers": [
                        {
                            "env": [
                                {
                                    "name": "SCOPE",
                                    "value": "acid-minimal-cluster"
                                },
                                {
                                    "name": "PGROOT",
                                    "value": "/home/postgres/pgdata/pgroot"
                                },
                                {
                                    "name": "POD_IP",
                                    "valueFrom": {
                                        "fieldRef": {
                                            "apiVersion": "v1",
                                            "fieldPath": "status.podIP"
                                        }
                                    }
                                },
                                {
                                    "name": "POD_NAMESPACE",
                                    "valueFrom": {
                                        "fieldRef": {
                                            "apiVersion": "v1",
                                            "fieldPath": "metadata.namespace"
                                        }
                                    }
                                },
                                {
                                    "name": "PGUSER_SUPERUSER",
                                    "value": "postgres"
                                },
                                {
                                    "name": "KUBERNETES_SCOPE_LABEL",
                                    "value": "cluster-name"
                                },
                                {
                                    "name": "KUBERNETES_ROLE_LABEL",
                                    "value": "spilo-role"
                                },
                                {
                                    "name": "PGPASSWORD_SUPERUSER",
                                    "valueFrom": {
                                        "secretKeyRef": {
                                            "key": "password",
                                            "name": "postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do"
                                        }
                                    }
                                },
                                {
                                    "name": "PGUSER_STANDBY",
                                    "value": "standby"
                                },
                                {
                                    "name": "PGPASSWORD_STANDBY",
                                    "valueFrom": {
                                        "secretKeyRef": {
                                            "key": "password",
                                            "name": "standby.acid-minimal-cluster.credentials.postgresql.acid.zalan.do"
                                        }
                                    }
                                },
                                {
                                    "name": "PAM_OAUTH2",
                                    "value": "https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees"
                                },
                                {
                                    "name": "HUMAN_ROLE",
                                    "value": "zalandos"
                                },
                                {
                                    "name": "ENABLE_WAL_PATH_COMPAT",
                                    "value": "true"
                                },
                                {
                                    "name": "PGVERSION",
                                    "value": "13"
                                },
                                {
                                    "name": "KUBERNETES_LABELS",
                                    "value": "{\"application\":\"spilo\"}"
                                },
                                {
                                    "name": "SPILO_CONFIGURATION",
                                    "value": "{\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}}"
                                },
                                {
                                    "name": "DCS_ENABLE_KUBERNETES_API",
                                    "value": "true"
                                }
                            ],
                            "image": "registry.opensource.zalan.do/acid/spilo-14:2.1-p6",
                            "imagePullPolicy": "IfNotPresent",
                            "name": "postgres",
                            "ports": [
                                {
                                    "containerPort": 8008,
                                    "protocol": "TCP"
                                },
                                {
                                    "containerPort": 5432,
                                    "protocol": "TCP"
                                },
                                {
                                    "containerPort": 8080,
                                    "protocol": "TCP"
                                }
                            ],
                            "readinessProbe": {
                                "failureThreshold": 3,
                                "httpGet": {
                                    "path": "/readiness",
                                    "port": 8008,
                                    "scheme": "HTTP"
                                },
                                "initialDelaySeconds": 6,
                                "periodSeconds": 10,
                                "successThreshold": 1,
                                "timeoutSeconds": 5
                            },
                            "resources": {
                                "limits": {
                                    "cpu": "1",
                                    "memory": "500Mi"
                                },
                                "requests": {
                                    "cpu": "100m",
                                    "memory": "100Mi"
                                }
                            },
                            "securityContext": {
                                "allowPrivilegeEscalation": true,
                                "privileged": false,
                                "readOnlyRootFilesystem": false
                            },
                            "terminationMessagePath": "/dev/termination-log",
                            "terminationMessagePolicy": "File",
                            "volumeMounts": [
                                {
                                    "mountPath": "/home/postgres/pgdata",
                                    "name": "pgdata"
                                },
                                {
                                    "mountPath": "/dev/shm",
                                    "name": "dshm"
                                }
                            ]
                        }
                    ],
                    "dnsPolicy": "ClusterFirst",
                    "restartPolicy": "Always",
                    "schedulerName": "default-scheduler",
                    "securityContext": {},
                    "serviceAccount": "postgres-pod",
                    "serviceAccountName": "postgres-pod",
                    "terminationGracePeriodSeconds": 300,
                    "volumes": [
                        {
                            "emptyDir": {
                                "medium": "Memory"
                            },
                            "name": "dshm"
                        }
                    ]
                }
            },
            "updateStrategy": {
                "type": "OnDelete"
            },
            "volumeClaimTemplates": [
                {
                    "apiVersion": "v1",
                    "kind": "PersistentVolumeClaim",
                    "metadata": {
                        "creationTimestamp": null,
                        "labels": {
                            "application": "spilo",
                            "cluster-name": "acid-minimal-cluster",
                            "team": "acid"
                        },
                        "name": "pgdata"
                    },
                    "spec": {
                        "accessModes": [
                            "ReadWriteOnce"
                        ],
                        "resources": {
                            "requests": {
                                "storage": "1Gi"
                            }
                        },
                        "volumeMode": "Filesystem"
                    },
                    "status": {
                        "phase": "Pending"
                    }
                }
            ]
        },
        "status": {
            "availableReplicas": 2,
            "collisionCount": 0,
            "currentReplicas": 2,
            "currentRevision": "acid-minimal-cluster-6f87b7d65d",
            "observedGeneration": 3,
            "readyReplicas": 2,
            "replicas": 2,
            "updateRevision": "acid-minimal-cluster-6f87b7d65d",
            "updatedReplicas": 2
        }
    }
}
@gravesm
Copy link
Member

gravesm commented Aug 23, 2022

@haroonb Thanks for the bug report. I'm able to reproduce this. In this case, the problem is because the wait condition is requiring that the StatefulSet's updateStrategy is RollingUpdate. If we just remove this check, however, it would break the wait when updating the container images for the StatefulSet, so we'll have to figure out how to support both cases.

@gravesm gravesm added type/bug Something isn't working verified The issue is reproduced jira labels Aug 23, 2022
@mandar242 mandar242 mentioned this issue Feb 2, 2023
Merged
softwarefactory-project-zuul bot pushed a commit that referenced this issue Feb 6, 2023
@mandar242
k8s_scale - handle scaling StatefulSets with 'updateStrategy=OnDelete' (
deaf8ee 
#579)

k8s_scale - handle scaling StatefulSets with 'updateStrategy=OnDelete'

SUMMARY

Likely Fixes #503

Handle scaling StatefulSets with 'updateStrategy=OnDelete'
ISSUE TYPE


Bugfix Pull Request

COMPONENT NAME

k8s_scale
ADDITIONAL INFORMATION

Reviewed-by: Mike Graves <mgraves@redhat.com>
Reviewed-by: Bikouo Aubin <None>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
jira type/bug Something isn't working verified The issue is reproduced
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

k8s_scale - handle scaling StatefulSets with 'updateStrategy=OnDelete' mandar242/kubernetes.core
2 participants
@gravesm @haroonb