Wait until deploy completes #58

chuhlomin · 2017-12-11T18:48:21Z

Related to #26

Example usage:

    rollout_check: [bodega-server, bodega-server-envoy]
    rollout_timeout: 300

Example output:

$ timeout -t 300 kubectl rollout status deploy bodega-server --namespace pubp-dev
Waiting for rollout to finish: 2 old replicas are pending termination...
Waiting for rollout to finish: 2 old replicas are pending termination...
Waiting for rollout to finish: 2 old replicas are pending termination...
Waiting for rollout to finish: 2 old replicas are pending termination...
deployment "bodega-server" successfully rolled out

There are no checks for deamonset or statefulset, but it would be easy to add this functionality.

jrynyt

thank you for submitting!

jrynyt · 2017-12-13T14:25:11Z

Dockerfile


 # kubectl 1.7.0
-#FROM google/cloud-sdk:163.0.0-alpine
+FROM google/cloud-sdk:163.0.0-alpine


they're up to 182 now; any reason to not use that? (I don't know the history of why these are both here)

I didn't find a build for 182, so I changed it to 181.
https://hub.docker.com/r/google/cloud-sdk/builds/
I don't know the history of this either.

jrynyt · 2017-12-13T14:26:12Z

main.go

@@ -126,6 +128,17 @@ func wrapMain() error {
 			Usage:  "Git tag",
 			EnvVar: "DRONE_TAG",
 		},
+		cli.StringSliceFlag{
+			Name:   "rollout_check",


can this be called wait_deployments or something like that, to make it obvious that the values are Deployments?

It's not necessary deployments, it could be also daemonsets or statefulsets.

not the way you've set it up and documented it here

I haven't tried, but I think this should work:

rollout_check: [gateway-server, deployment/gateway-server, daemonsets/some-set]

that sounds great, but deployment is hardcoded in the command below (L439)

Right, I see. I'll do it for deployment only for now.

jrynyt · 2017-12-13T14:26:30Z

main.go

+			EnvVar: "PLUGIN_ROLLOUT_CHECK",
+		},
+		cli.IntFlag{
+			Name:   "rollout_timeout",


similarly, wait_seconds

might want to mention that exceeding the timeout fails the build

jrynyt · 2017-12-13T14:46:43Z

main.go

+			"rollout",
+			"status",
+			"deploy",
+			deployment,


I think the output from running these in series could be confusing -- I would wonder why the other deployments aren't showing up. maybe adding & to the end of the command, or otherwise running in parallel, would be more intuitive.

We should tackle this in a follow up.

Adding & would run them in parallel, however mix up the log messages (that might not matter if the end result includes the name deployment "x" successfully rolled out).

IMO checking that the deployment succeeded should occur in another step.
This allows the user to configure run as many steps as there are deployments, and group these steps so they run in parallel.

Of course the downside of this is the number of resources being rolled out in each kind of event would need to be known ahead of time (unless that step was a configurable plugin that could parse .kube.yml, or something).

Created issue to track here: #64.

jrynyt · 2017-12-13T14:49:12Z

main.go

+		if rolloutTimeout == 0 {
+			err = runner.Run(kubectlCmd, command[3:]...)
+		} else {
+			err = runner.Run(timeoutCmd, command...)


not that big of a deal, but I find all this handling of the command a little confusing.

chuhlomin · 2017-12-14T16:23:59Z

Updated code.
Example usage:

    wait_deployments: [bodega-server, bodega-server-envoy]
    wait_seconds: 300

Example output:

[1/2] Waiting till rollout completes for bodega-server

$ timeout -t 300 kubectl rollout status deployment bodega-server --namespace pubp-dev
deployment "bodega-server" successfully rolled out

[2/2] Waiting till rollout completes for bodega-server-envoy

$ timeout -t 300 kubectl rollout status deployment bodega-server-envoy --namespace pubp-dev
deployment "bodega-server-envoy" successfully rolled out

jrynyt

this is great, thanks again.

would you mind adding this info to DOCS.md?

chuhlomin · 2017-12-18T19:00:16Z

Sure, updated

jrynyt · 2017-12-27T17:26:08Z

main.go

+			counterProgress = fmt.Sprintf("[%d/%d]", counter + 1, waitDeploymentsCount)
+		}
+
+		log(fmt.Sprintf("%sWaiting till rollout completes for %s\n", counterProgress, deployment))


"Waiting until rollout completes for Deployment 1/2"

Otherwise this will get conflated with the output of the command itself

jrynyt · 2017-12-27T17:29:18Z

DOCS.md

@@ -13,6 +13,8 @@ The following parameters are used to configure this plugin:
 * *optional* `namespace` - Kubernetes namespace to operate in (defaults to `default`)
 * *optional* `template` - Kubernetes manifest template (defaults to `.kube.yml`)
 * *optional* `secret_template` - Kubernetes [_Secret_ resource](http://kubernetes.io/docs/user-guide/secrets/) manifest template (defaults to `.kube.sec.yml`)
+* *optional* `wait_deployments` - list of deployments to check after apply


List of Deployments to wait for successful rollout, using kubectl rollout status

jrynyt · 2017-12-27T17:29:41Z

DOCS.md

@@ -13,6 +13,8 @@ The following parameters are used to configure this plugin:
 * *optional* `namespace` - Kubernetes namespace to operate in (defaults to `default`)
 * *optional* `template` - Kubernetes manifest template (defaults to `.kube.yml`)
 * *optional* `secret_template` - Kubernetes [_Secret_ resource](http://kubernetes.io/docs/user-guide/secrets/) manifest template (defaults to `.kube.sec.yml`)
+* *optional* `wait_deployments` - list of deployments to check after apply
+* *optional* `wait_seconds` - if `wait_deployments` is set, maximum number of seconds for rollout (the exceeded timeout сauses the build to fail)


number of seconds to wait before failing the build

chuhlomin · 2018-01-18T20:55:56Z

Is there anything else I could do?

jrynyt · 2018-01-22T14:34:52Z

you mean to get this merged? I added some more reviewers. Note that I'm pretty sure this will only apply to drone.dv.nyt.net; not the old ones.

tonglil

I am approving for people who want this functionality now.

Merging to develop will make this available in nytimes/drone-gke:develop.

Will need to tag a 0.7.x release for this as well.

tonglil · 2018-01-24T17:18:12Z

Dockerfile

@@ -1,8 +1,8 @@
 # kubectl 1.6.6
-FROM google/cloud-sdk:161.0.0-alpine
+#FROM google/cloud-sdk:161.0.0-alpine


can you delete these two lines?

tonglil · 2018-01-24T17:18:20Z

Dockerfile

@@ -1,8 +1,8 @@
 # kubectl 1.6.6
-FROM google/cloud-sdk:161.0.0-alpine
+#FROM google/cloud-sdk:161.0.0-alpine

 # kubectl 1.7.0


Can you update this version number?

tonglil · 2018-01-24T19:46:38Z

Edit on my comment above: will need to tag a 0.8.* release from develop post merge for this since it has new functionality.

chuhlomin added 2 commits November 10, 2017 15:28

Added rollout_check plugin argument

370b908

Added rollout_timeout plugin argument

e7eae61

jrynyt suggested changes Dec 13, 2017

View reviewed changes

Renamed parameters to wait_deployments and wait_seconds nytimes#58

5b7c5a7

Updated usage comment for wait_seconds flag nytimes#58

d49ac25

jrynyt approved these changes Dec 15, 2017

View reviewed changes

chuhlomin added 2 commits December 15, 2017 11:34

Updated DOCS.md nytimes#58

6dd5e41

Print counter only when waiting for more than one deployment nytimes#58

63ab2e0

jrynyt force-pushed the develop branch from 78d8b11 to 2d6a81b Compare December 27, 2017 16:41

jrynyt reviewed Dec 27, 2017

View reviewed changes

Updated DOCS.md and console output nytimes#58

cb7a75b

jrynyt force-pushed the develop branch from 2d6a81b to f9d6973 Compare January 3, 2018 17:32

jrynyt requested review from balser, msuterski and tonglil January 22, 2018 14:34

tonglil reviewed Jan 24, 2018

View reviewed changes

Merge branch 'develop' into develop

abfdd73

tonglil approved these changes Jan 24, 2018

View reviewed changes

tonglil added the version/0.5+ label Jan 24, 2018

tonglil merged commit 41313cf into nytimes:develop Jan 24, 2018

tonglil mentioned this pull request Jan 25, 2018

Bump version #65

Merged

montmanu mentioned this pull request Oct 29, 2020

Replace custom rollout timeout implementation with kubectl native alternative #136

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wait until deploy completes #58

Wait until deploy completes #58

chuhlomin commented Dec 11, 2017 •

edited

Loading

jrynyt left a comment

jrynyt Dec 13, 2017

chuhlomin Dec 14, 2017

jrynyt Dec 13, 2017

chuhlomin Dec 13, 2017

jrynyt Dec 13, 2017

chuhlomin Dec 13, 2017

jrynyt Dec 14, 2017

chuhlomin Dec 14, 2017

jrynyt Dec 13, 2017

jrynyt Dec 13, 2017

tonglil Jan 24, 2018 •

edited

Loading

jrynyt Dec 13, 2017

chuhlomin commented Dec 14, 2017

jrynyt left a comment

chuhlomin commented Dec 18, 2017

jrynyt Dec 27, 2017

jrynyt Dec 27, 2017

jrynyt Dec 27, 2017 •

edited

Loading

chuhlomin commented Jan 18, 2018

jrynyt commented Jan 22, 2018

tonglil left a comment •

edited

Loading

tonglil Jan 24, 2018

tonglil Jan 24, 2018

tonglil commented Jan 24, 2018 •

edited

Loading

Wait until deploy completes #58

Wait until deploy completes #58

Conversation

chuhlomin commented Dec 11, 2017 • edited Loading

jrynyt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonglil Jan 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chuhlomin commented Dec 14, 2017

jrynyt left a comment

Choose a reason for hiding this comment

chuhlomin commented Dec 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrynyt Dec 27, 2017 • edited Loading

Choose a reason for hiding this comment

chuhlomin commented Jan 18, 2018

jrynyt commented Jan 22, 2018

tonglil left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonglil commented Jan 24, 2018 • edited Loading

chuhlomin commented Dec 11, 2017 •

edited

Loading

tonglil Jan 24, 2018 •

edited

Loading

jrynyt Dec 27, 2017 •

edited

Loading

tonglil left a comment •

edited

Loading

tonglil commented Jan 24, 2018 •

edited

Loading