use kubectl-with-retry on pause & resume #26243

metral · 2016-05-25T04:52:11Z

attempts to fix #25645 by using kubectl-with-retry on rollout {pause,resume} (resume is for safe measures) instead of kubectl directly, as is done with other rollout {pause,resume} tests in this same script.

k8s-bot · 2016-05-25T04:52:30Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-05-25T04:53:11Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-05-25T04:54:12Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

eparis · 2016-05-25T13:05:43Z

ok to test

metral · 2016-05-25T14:38:47Z

The e2e and Node e2e tests are hitting the following errors:

ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-jenkins-pull/global/firewalls/e2e-gce-agent-pr-3-0-minion-e2e-gce-agent-pr-3-0-http-alt' was not found
ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-jenkins-pull/global/firewalls/e2e-gce-agent-pr-3-0-minion-e2e-gce-agent-pr-3-0-nodeports' was not found
Bringing down cluster using provider: gce

ERROR: gcloud crashed (OSError): [Errno 39] Directory not empty: '/usr/local/share/google/google-cloud-sdk.staging/lib/googlecloudsdk/third_party/apis'

If you would like to report this issue, please run the following command:
  gcloud feedback
Project: kubernetes-jenkins-pull
Zone: us-central1-f
INSTANCE_GROUPS=
NODE_NAMES=
Bringing down cluster
ERROR: gcloud failed to load: No module named argcomplete

WARNING: You are creating a legacy network. Using --mode=legacy will be required in future releases.
ERROR: (gcloud.compute.networks.create) Some requests did not succeed:
 - Quota 'ROUTES' exceeded.  Limit: 170.0

2016/05/25 06:28:42 e2e.go:206: Error running up: exit status 1
2016/05/25 06:28:42 e2e.go:202: Step 'up' finished in 47.621947144s
2016/05/25 06:28:42 e2e.go:114: Error starting e2e cluster. Aborting.

eparis · 2016-05-25T16:38:12Z

@k8s-bot e2e test this flake: #26271

janetkuo · 2016-05-25T17:12:52Z

hack/test-cmd.sh

@@ -1087,13 +1087,11 @@ __EOF__
  kube::test::get_object_assert deployment "{{range.items}}{{$deployment_image_field}}:{{end}}" "${IMAGE_NGINX}:${IMAGE_NGINX}:"
  kube::test::if_has_string "${output_message}" "Object 'Kind' is missing"
  ## Pause the deployment
-  output_message=$(! kubectl rollout pause -f hack/testdata/recursive/deployment --recursive 2>&1 "${kube_flags[@]}")


Would you change the title comment of this test too (L1077)? These tests are about "--recursive" and the "rollout" commands, but not about "Rollback a deployment". It's somewhat misleading. Changing each sub-title would help too (maybe add "recursively" to the end of each sub-title, and mention that we expect error message "Object 'Kind' is missing").

janetkuo · 2016-05-25T17:22:18Z

hack/test-cmd.sh

  kube::test::get_object_assert deployment "{{range.items}}{{.spec.paused}}:{{end}}" "true:true:"
-  kube::test::if_has_string "${output_message}" "Object 'Kind' is missing"


I believe we still want this check?

We do, but I removed it as it was the only idea I had for a solution to attempt to triage the flake quickly by using kubectl-with-retry instead of kubectl.

However, in doing so, kubectl-with-retry directs stderr to a file that is removed once it is done executing, and does not allow me to review it for this missing object as I was doing

We can have a retry function with customized error message check, or we can pass the expected error message to that retry function. WDYT?

Do we still need 2>&1 when we're using kubectl-with-retry?

Maybe creating a new retry function with your own check is easier, and we can add a comment saying something like "remove this function once #20437 is fixed"

metral · 2016-05-25T17:58:43Z

The e2e test is hitting flake #26210

janetkuo · 2016-05-25T18:21:38Z

@k8s-bot e2e test this issue: #IGNORE

metral · 2016-05-25T19:13:05Z

@janetkuo I've gone ahead and addressed your feedback:

cleared up test title comments to describe the tests more effectively
removed 2>&1 from kubectl-with-retry as it wasn't needed
modified kubectl-with-retry slightly to allow me to preserve the error file for review/usage in my tests, allowing me to put the checks that I originally took out, back in

PTAL

janetkuo · 2016-05-25T20:36:11Z

Thanks, looks good now. Please squash and I'll apply the tag.

metral · 2016-05-25T20:38:09Z

@janetkuo all set, thanks!

k8s-github-robot · 2016-05-26T01:55:01Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-05-26T02:33:24Z

GCE e2e build/test passed for commit 54e6d23.

k8s-github-robot · 2016-05-26T02:49:45Z

Automatic merge from submit-queue

googlebot added the cla: yes label May 25, 2016

metral mentioned this pull request May 25, 2016

test-cmd flake: kubectl rollout pause deployments --recursive #25645

Closed

k8s-github-robot assigned eparis May 25, 2016

k8s-github-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note-label-needed labels May 25, 2016

janetkuo reviewed May 25, 2016
View reviewed changes

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 25, 2016

use kubectl-with-retry on pause & resume

54e6d23

metral force-pushed the fix-pause branch from 328af21 to 54e6d23 Compare May 25, 2016 20:38

janetkuo added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2016

k8s-github-robot added the needs-ok-to-merge label May 25, 2016

janetkuo added ok-to-merge and removed needs-ok-to-merge labels May 25, 2016

k8s-github-robot merged commit 128e7f1 into kubernetes:master May 26, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use kubectl-with-retry on pause & resume #26243

use kubectl-with-retry on pause & resume #26243

metral commented May 25, 2016

k8s-bot commented May 25, 2016

k8s-bot commented May 25, 2016

k8s-bot commented May 25, 2016

eparis commented May 25, 2016

metral commented May 25, 2016

eparis commented May 25, 2016

janetkuo May 25, 2016

metral May 25, 2016

janetkuo May 25, 2016

metral May 25, 2016 •

edited

Loading

janetkuo May 25, 2016 •

edited

Loading

janetkuo May 25, 2016

janetkuo May 25, 2016 •

edited

Loading

metral commented May 25, 2016

janetkuo commented May 25, 2016

metral commented May 25, 2016 •

edited

Loading

janetkuo commented May 25, 2016

metral commented May 25, 2016

k8s-github-robot commented May 26, 2016

k8s-bot commented May 26, 2016

k8s-github-robot commented May 26, 2016

		kube::test::get_object_assert deployment "{{range.items}}{{.spec.paused}}:{{end}}" "true:true:"
		kube::test::if_has_string "${output_message}" "Object 'Kind' is missing"

use kubectl-with-retry on pause & resume #26243

use kubectl-with-retry on pause & resume #26243

Conversation

metral commented May 25, 2016

k8s-bot commented May 25, 2016

k8s-bot commented May 25, 2016

k8s-bot commented May 25, 2016

eparis commented May 25, 2016

metral commented May 25, 2016

eparis commented May 25, 2016

janetkuo May 25, 2016

Choose a reason for hiding this comment

metral May 25, 2016

Choose a reason for hiding this comment

janetkuo May 25, 2016

Choose a reason for hiding this comment

metral May 25, 2016 • edited Loading

Choose a reason for hiding this comment

janetkuo May 25, 2016 • edited Loading

Choose a reason for hiding this comment

janetkuo May 25, 2016

Choose a reason for hiding this comment

janetkuo May 25, 2016 • edited Loading

Choose a reason for hiding this comment

metral commented May 25, 2016

janetkuo commented May 25, 2016

metral commented May 25, 2016 • edited Loading

janetkuo commented May 25, 2016

metral commented May 25, 2016

k8s-github-robot commented May 26, 2016

k8s-bot commented May 26, 2016

k8s-github-robot commented May 26, 2016

metral May 25, 2016 •

edited

Loading

janetkuo May 25, 2016 •

edited

Loading

janetkuo May 25, 2016 •

edited

Loading

metral commented May 25, 2016 •

edited

Loading