Refine deployment rollouts #1222

stanislav-zaprudskiy · 2023-02-07T15:29:53Z

SUMMARY

This is a groundwork which originated in and got split out from #1193 as being beneficial independently from that PR. It addresses some unintended behavior and edge cases during Deployment rollouts, making it more robust and reliable. It's better to be reviewed on a per commit basis, with details and rationale available in the corresponding commit messages.

ISSUE TYPE

Bug, Docs Fix or other nominal change

ADDITIONAL INFORMATION

There are cases when having a new Deployment may be taking above the default timeout of 120s. For instance, when a Deployment has multiple replicas, and each replica starts on a separate node, and the Deployment specifies new images, then just pulling these new images for each replica may be taking above the default timeout of 120s. Having the default time multiplied by the number of replicas should provide generally enough time for all replicas to start

Proper waiting is already performed earlier during Deplyment{apply: yes, wait: yes} - https://github.com/ansible-collections/kubernetes.core/blob/e6ac87409830b6c698b05dba875cca87d45ea761/plugins/module_utils/k8s/waiter.py#L27. And also not every Deployment change produces new RS/Pods. For example, changing Deployment labels won't cause new rollout, but will cause `until` loop to be invoked unnecessarily (when replicas=1).

… and `deletionTimestamp` Do not consider Pods marked for deletion when calculating tower_pod to address replicas scale down case - where normally Pods spawned recently are being taken for removal. As well as the case when operator kicked off but some old replicas are still terminating. Respect `creationTimestamp` so to make sure that the newest Pod is taken after Deployment application, in which case multiple RS Pods (from old RS and new RS) could be running simultaneously while the rollout is happening.

With the previous approach, not all associated (mounted) CM/Secrets changes caused the Deployment to be rolled out, but also the Deployment could have been rolled out unnecessary during e.g. Ingress or Service changes (which do not require Pod restarts). Previously existing Pod removal (state: absent) was not complete as other pods continued to exist, but also is not needed with this commit change due to added Pods annotations. The added Deployment Pod annotations now cause the new ReplicaSet version to be rolled out, effectively causing replacement of the previously existing Pods in accordance with the deployment `strategy` (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#deploymentstrategy-v1-apps, `RollingUpdate`) whenever there is a change in the associated CMs or Secrets referenced in annotations. This implementation is quite standard and widely used for Helm workflows - https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments

fosterseth · 2023-02-08T19:05:53Z

@stanislav-zaprudskiy thanks for splitting out this work from the other PR. We'll get some eyes on this PR shortly

rooftopcellist · 2023-02-15T19:25:54Z

roles/installer/tasks/resources_configuration.yml

@@ -13,9 +13,17 @@
      - status.phase=Running
  register: tower_pod

+- name: Set the resource pod as a variable.


This will prevent us from grabbing a pod that is in terminating state. It also ensures that only one pod is grabbed (the oldest).

Just adding the corresponding commit message in case it got missed

commit b3a7436

Make AWX Pod variable to be calculated respecting `creationTimestamp` and `deletionTimestamp` Do not consider Pods marked for deletion when calculating tower_pod to address replicas scale down case - where normally Pods spawned recently are being taken for removal. As well as the case when operator kicked off but some old replicas are still terminating. Respect `creationTimestamp` so to make sure that the newest Pod is taken after Deployment application, in which case multiple RS Pods (from old RS and new RS) could be running simultaneously while the rollout is happening.

I've also encountered a couple of cases when Terminating pod was picked up, and running awx-manage command on it during the later tasks was not possible (the pod was gone already, etc).

Here indeed the oldest pod is taken, while later in the code the newest (the one most recently created) is used.

rooftopcellist · 2023-02-15T19:30:08Z

roles/installer/tasks/resources_configuration.yml

@@ -210,21 +250,10 @@
    apply: yes
    definition: "{{ lookup('template', 'deployments/deployment.yaml.j2') }}"
    wait: yes
+    wait_timeout: "{{ 120 * replicas or 120 }}"


@stanislav-zaprudskiy could you expand on why you added this?

Adding the corresponding commit message just in case it got missed

commit e589ceb

When applying Deployment wait up to (timeout * replicas) There are cases when having a new Deployment may be taking above the default timeout of 120s. For instance, when a Deployment has multiple replicas, and each replica starts on a separate node, and the Deployment specifies new images, then just pulling these new images for each replica may be taking above the default timeout of 120s. Having the default time multiplied by the number of replicas should provide generally enough time for all replicas to start

The corresponding parameter wait: yes (already provided) causes the task to wait until Deployment pods are ready, and with the default 120 seconds it could fail depending on the overall k8s and AWX configurations, causing the operator run to fail and be started again. Starting a new run won't solve the problem however, as it would run into the same timeout issue again and again until the new pods are ready - so by increasing maximum waiting time originally it saves from further failures and restarts.

Just to add, also the Deployment's rollout strategy configuration could increase the time for having the new pods ready.

There could be cases when scheduling new pods may not be possible (e.g. due to lack of resources, or wrong image configuration, etc) - but the operator would stuck (unnecessary) waiting. Having lower timeout value would make users aware of the problem earlier - but in multi-replica AWX configurations running in multi-node clusters where image caches aren't generally available, the lower timeout values provide too much false-positive failures of operator runs.

rooftopcellist · 2023-02-15T19:30:31Z

roles/installer/tasks/secret_key_configuration.yml

@@ -40,10 +40,10 @@

 - name: Set secret key secret
  set_fact:
-    __secret_key_secret: '{{ _generated_secret_key["resources"] | default([]) | length | ternary(_generated_secret_key, _secret_key_secret)  }}'
+    secret_key: '{{ _generated_secret_key["resources"] | default([]) | length | ternary(_generated_secret_key, _secret_key_secret)  }}'


I want to do some testing around this before merging. cc @TheRealHaoLiu

rooftopcellist · 2023-02-28T21:15:52Z

roles/installer/templates/deployments/deployment.yaml.j2

+    "secrets/app_credentials",
+    "storage/persistent",
+  ] %}
+        checksum-{{ template | replace('/', '-') }}: "{{ lookup('template', template + '.yaml.j2') | md5 }}"


❤️ This is a great trick! This is an elegant solution to the issue of deployments not being cycled when changes are made to the ConfigMap. @TheRealHaoLiu is impressed too.

rooftopcellist · 2023-02-28T21:19:09Z

@stanislav-zaprudskiy Thank you for this PR, it is exemplary. 🏆
I have learned a lot from your contributions, and hope to learn more!

stanislav-zaprudskiy added 4 commits February 7, 2023 11:41

github-actions bot added the community label Feb 7, 2023

Fix lint warnings

f042cb3

stanislav-zaprudskiy mentioned this pull request Feb 7, 2023

AWX: Add termination_grace_period_seconds #1193

Merged

fosterseth self-requested a review February 8, 2023 19:05

TheRealHaoLiu assigned TheRealHaoLiu, rooftopcellist and fosterseth Feb 14, 2023

rooftopcellist reviewed Feb 15, 2023

View reviewed changes

TheRealHaoLiu merged commit f042cb3 into ansible:devel Feb 28, 2023

rooftopcellist reviewed Feb 28, 2023

View reviewed changes

rooftopcellist mentioned this pull request Mar 1, 2023

Helm upgrade is not detecting AWX kind updates #1239

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refine deployment rollouts #1222

Refine deployment rollouts #1222

stanislav-zaprudskiy commented Feb 7, 2023

fosterseth commented Feb 8, 2023

rooftopcellist Feb 15, 2023

stanislav-zaprudskiy Feb 17, 2023

rooftopcellist Feb 15, 2023

stanislav-zaprudskiy Feb 17, 2023

rooftopcellist Feb 15, 2023

rooftopcellist Feb 28, 2023 •

edited

rooftopcellist commented Feb 28, 2023

Refine deployment rollouts #1222

Refine deployment rollouts #1222

Conversation

stanislav-zaprudskiy commented Feb 7, 2023

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

fosterseth commented Feb 8, 2023

rooftopcellist Feb 15, 2023

Choose a reason for hiding this comment

stanislav-zaprudskiy Feb 17, 2023

Choose a reason for hiding this comment

rooftopcellist Feb 15, 2023

Choose a reason for hiding this comment

stanislav-zaprudskiy Feb 17, 2023

Choose a reason for hiding this comment

rooftopcellist Feb 15, 2023

Choose a reason for hiding this comment

rooftopcellist Feb 28, 2023 • edited

Choose a reason for hiding this comment

rooftopcellist commented Feb 28, 2023

rooftopcellist Feb 28, 2023 •

edited