Graduate DS maxSurge to beta #2665

ravisantoshgudimetla · 2021-04-30T22:28:45Z

Most of the questions in the PRR section seems answered and if needed they can be answered when GA'ing.

JamesLaverack · 2021-05-06T01:55:11Z

xref #1591

keps/prod-readiness/sig-apps/1591.yaml

ehashman

Hi @ravisantoshgudimetla,

The required PRR sections for beta are not currently filled out. Please complete them so I can review. They are:

Rollout, Upgrade and Rollback Planning
Monitoring Requirements
Dependencies
Scalability (completed)
Troubleshooting

ehashman · 2021-05-06T23:16:30Z

keps/sig-apps/1591-daemonset-surge/kep.yaml

@@ -27,7 +30,7 @@ feature-gates:
      - kube-controller-manager
 disable-supported: true
 stage: "alpha"


This should read beta, right?

So, this should be intended state or current state? I can change this to beta, if it has to be intended state. To be clear, I thought it has to be changed to beta once the implementation merges :(

Yes, this should be the intended state.

ravisantoshgudimetla

Thank you for the initial feedback @ehashman

_This section must be completed when targeting beta graduation to a release._. This statement confused me, I was thinking this has to be done when we are targeting from beta to GA. I'll fill those values and get back to you. Apologies for lack of understanding on my part.

ehashman · 2021-05-07T17:26:36Z

No problem, looking forward to your responses.

ravisantoshgudimetla

@ehashman - I added content for the following sub-sections

Rollout, Upgrade and Rollback Planning
Monitoring Requirements
Dependencies
Troubleshooting
PTAL and let me know, if I am missing any other info

soltysh · 2021-05-10T20:24:01Z

keps/sig-apps/1591-daemonset-surge/README.md

-  - 99% percentile over day of absolute value from (job creation time minus expected
-    job creation time) for cron job <= 10%
-  - 99,9% of /health requests per day finish with 200 code
+  None


Even though the SLI is rather "manual" we should be guarantee some reasonable SLO for it.

I changed it now.

soltysh · 2021-05-10T20:24:37Z

keps/sig-apps/1591-daemonset-surge/README.md

@@ -328,18 +309,10 @@ details). For now, we leave it here.
 _This section must be completed when targeting beta graduation to a release._

 * **How does this feature react if the API server and/or etcd is unavailable?**
+This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events for StatefulSets.


get events nor provide updates.

Isn't update an event too. ;). I changed the sentence to provide more clarity.

ehashman · 2021-05-11T01:05:46Z

keps/sig-apps/1591-daemonset-surge/README.md


 * **What specific metrics should inform a rollback?**
+  None


How can you tell if the feature is failing then?

ehashman · 2021-05-11T01:06:16Z

keps/sig-apps/1591-daemonset-surge/README.md

-  Describe manual testing that was done and the outcomes.
-  Longer term, we may want to require automated upgrade/rollback tests, but we
-  are missing a bunch of machinery and tooling and can't do that now.
+  Manually tested. No issues were found.


Can you please list the specific scenarios that were/will be tested?

ehashman · 2021-05-11T01:07:37Z

keps/sig-apps/1591-daemonset-surge/README.md

-  Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
-  checking if there are objects with field X set) may be a last resort. Avoid
-  logs or events for this purpose.
+  By checking the StatefulSet's RollingUpdate strategy.


Can you provide the exact key in the spec to check or an example?

ehashman · 2021-05-11T01:08:14Z

keps/sig-apps/1591-daemonset-surge/README.md

-    - Details:
+  - [x] Other (treat as last resort)
+    - Details: The number of pods that are created above the desired amount of pods during an update when this feature is enabled can be compared to maxSurge value available in
+    the StatefulSet definition. This can be used to determine the health of this feature


There are existing metrics in Kubernetes that describe this, can you please list them under the metrics section?

ehashman · 2021-05-11T01:10:45Z

keps/sig-apps/1591-daemonset-surge/README.md

-  - 99% percentile over day of absolute value from (job creation time minus expected
-    job creation time) for cron job <= 10%
-  - 99,9% of /health requests per day finish with 200 code
+  All the surge pods created should be within the value(% or number) of maxSurge field provided 99.99% of the time.


I don't understand this SLO.

From the summary:

This will allow daemonset workloads to implement zero-downtime upgrades.

Thus, I would expect your SLOs to be targeted at high availability for daemonset workloads, and describe to cluster admins how to measure this.

ehashman · 2021-05-11T01:11:58Z

keps/sig-apps/1591-daemonset-surge/README.md

@@ -328,18 +309,10 @@ details). For now, we leave it here.
 _This section must be completed when targeting beta graduation to a release._

 * **How does this feature react if the API server and/or etcd is unavailable?**
+This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets.


What happens if it's mid-rollout and this occurs? How can someone handle that failure?

ehashman · 2021-05-11T01:12:49Z

keps/sig-apps/1591-daemonset-surge/README.md

-      levels that could help debug the issue?
-      Not required until feature graduated to beta.
-    - Testing: Are there any tests for failure mode? If not, describe why.
+  None


This needs to be filled out. No feature in Kubernetes is failure-proof :) I'd like you to think of some possible ways that the feature could fail.

soltysh

A couple of nits.
/approve
from sig-apps pov

soltysh · 2021-05-11T11:50:31Z

keps/sig-apps/1591-daemonset-surge/README.md

-  Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
-  checking if there are objects with field X set) may be a last resort. Avoid
-  logs or events for this purpose.
+  By checking the StatefulSet's `.spec.strategy.rollingUpdate.maxSurge` field. The additional workload pods created should be respecting the value specified in the


Nit: this is about DaemonSet not StatefulSet 😉

soltysh · 2021-05-11T11:51:04Z

keps/sig-apps/1591-daemonset-surge/README.md

-    - Details:
+  - [x] Other (treat as last resort)
+    - Details: The number of pods that are created above the desired amount of pods during an update when this feature is enabled can be compared to maxSurge value available in
+    the StatefulSet definition. This can be used to determine the health of this feature.


s/StatefulSet/DaemonSet

soltysh · 2021-05-11T11:52:27Z

keps/sig-apps/1591-daemonset-surge/README.md

@@ -328,18 +317,24 @@ details). For now, we leave it here.
 _This section must be completed when targeting beta graduation to a release._

 * **How does this feature react if the API server and/or etcd is unavailable?**
+This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets. If the API server and/or etcd is unavailable during the mid-rollout, the featuregate would not be enabled and controller-manager wouldn't start since it cannot communicate with the API server


s/StatefulSet/DaemonSet

soltysh · 2021-05-11T11:54:11Z

keps/sig-cluster-lifecycle/kubeadm/2568-kubeadm-non-root-control-plane/README.md

@@ -93,6 +93,8 @@ tags, and then generate with `hack/update-toc.sh`.
  - [Updating the component manifests](#updating-the-component-manifests)


I don't expect any other changes in this PR.

soltysh

/lgtm

ehashman

/approve
for PRR

ehashman · 2021-05-11T18:47:05Z

keps/sig-apps/1591-daemonset-surge/README.md

+  All the surge pods created should be within the value(% or number) of maxSurge field provided 99.99% of the time. The additinal pods created should ensure that the workload
+  service is available 99.99% of time during updates.


4 9's is a very aggressive target, I don't think it makes sense as a universal SLO. But this is not blocking.

keps/sig-apps/1591-daemonset-surge/README.md

k8s-ci-robot · 2021-05-11T18:50:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ehashman, ravisantoshgudimetla, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [ehashman]
~~keps/sig-apps/OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ravisantoshgudimetla

@ehashman - Thank you for reviewing the PR from PRR side.

@soltysh - Thank you for reviewing the PR from sig-apps side.

kubernetes/kubernetes#101742 merged which made this feature beta. This is the companion docs PR for kubernetes/enhancements#2665

kubernetes/kubernetes#101742 merged which made this feature beta. This is the companion docs PR for kubernetes/enhancements#2665 Co-authored-by: Karen Bradshaw <kbhawkey@gmail.com>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 30, 2021

k8s-ci-robot requested review from mattfarina and soltysh April 30, 2021 22:28

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Apr 30, 2021

ravisantoshgudimetla mentioned this pull request May 5, 2021

Promote DS MaxSurge to beta kubernetes/kubernetes#101742

Merged

wojtek-t reviewed May 6, 2021

View reviewed changes

keps/prod-readiness/sig-apps/1591.yaml Outdated Show resolved Hide resolved

JamesLaverack mentioned this pull request May 6, 2021

DaemonSets should support MaxSurge to improve workload availability #1591

Closed

ravisantoshgudimetla force-pushed the promote-maxSurge-beta branch from c34cae3 to 3c3440e Compare May 6, 2021 21:43

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 6, 2021

ravisantoshgudimetla force-pushed the promote-maxSurge-beta branch 2 times, most recently from 6bcf815 to 7527dfe Compare May 6, 2021 21:51

ravisantoshgudimetla changed the title ~~[wip] Graduate DS maxSurge to beta~~ Graduate DS maxSurge to beta May 6, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2021

ehashman requested changes May 6, 2021

View reviewed changes

ehashman self-assigned this May 6, 2021

ravisantoshgudimetla commented May 7, 2021

View reviewed changes

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 7, 2021

ravisantoshgudimetla commented May 7, 2021

View reviewed changes

soltysh reviewed May 10, 2021

View reviewed changes

ravisantoshgudimetla force-pushed the promote-maxSurge-beta branch from 9b3c9ed to a389cb8 Compare May 10, 2021 20:40

ehashman requested changes May 11, 2021

View reviewed changes

ravisantoshgudimetla force-pushed the promote-maxSurge-beta branch from a389cb8 to 692049d Compare May 11, 2021 03:31

k8s-ci-robot added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label May 11, 2021

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 11, 2021

soltysh approved these changes May 11, 2021

View reviewed changes

Promote DS MaxSurge to beta

e749751

ravisantoshgudimetla force-pushed the promote-maxSurge-beta branch from 692049d to e749751 Compare May 11, 2021 12:39

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 11, 2021

soltysh approved these changes May 11, 2021

View reviewed changes

k8s-ci-robot assigned soltysh May 11, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 11, 2021

ehashman approved these changes May 11, 2021

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 11, 2021

k8s-ci-robot merged commit c09cc13 into kubernetes:master May 11, 2021

k8s-ci-robot added this to the v1.22 milestone May 11, 2021

ravisantoshgudimetla commented May 12, 2021

View reviewed changes

ravisantoshgudimetla mentioned this pull request May 19, 2021

Remove DaemonSetUpdateSurge feature from alpha kubernetes/test-infra#22263

Merged

ravisantoshgudimetla mentioned this pull request May 23, 2021

Introduce maxSurge as beta feature for DaemonSets kubernetes/website#28089

Closed

ravisantoshgudimetla mentioned this pull request Jun 7, 2021

Introduce maxSurge as beta feature for DaemonSets kubernetes/website#28319

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Graduate DS maxSurge to beta #2665

Graduate DS maxSurge to beta #2665

ravisantoshgudimetla commented Apr 30, 2021

JamesLaverack commented May 6, 2021

ehashman left a comment

ehashman May 6, 2021

ravisantoshgudimetla May 7, 2021 •

edited

Loading

ehashman May 7, 2021

ravisantoshgudimetla left a comment

ehashman commented May 7, 2021

ravisantoshgudimetla left a comment

soltysh May 10, 2021

ravisantoshgudimetla May 10, 2021

soltysh May 10, 2021

ravisantoshgudimetla May 10, 2021

ehashman May 11, 2021

ehashman May 11, 2021

ehashman May 11, 2021

ehashman May 11, 2021

ehashman May 11, 2021

ehashman May 11, 2021

ehashman May 11, 2021

soltysh left a comment

soltysh May 11, 2021

soltysh May 11, 2021

soltysh May 11, 2021

soltysh May 11, 2021

soltysh left a comment

ehashman left a comment

ehashman May 11, 2021

k8s-ci-robot commented May 11, 2021

ravisantoshgudimetla left a comment

		@@ -93,6 +93,8 @@ tags, and then generate with `hack/update-toc.sh`.
		- [Updating the component manifests](#updating-the-component-manifests)

		All the surge pods created should be within the value(% or number) of maxSurge field provided 99.99% of the time. The additinal pods created should ensure that the workload
		service is available 99.99% of time during updates.

Graduate DS maxSurge to beta #2665

Graduate DS maxSurge to beta #2665

Conversation

ravisantoshgudimetla commented Apr 30, 2021

JamesLaverack commented May 6, 2021

ehashman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravisantoshgudimetla May 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

ehashman commented May 7, 2021

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soltysh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soltysh left a comment

Choose a reason for hiding this comment

ehashman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 11, 2021

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

ravisantoshgudimetla May 7, 2021 •

edited

Loading