UPSTREAM: <carry>: retry etcd errors

[deads2k]

This retries all non-mutating requests for etcd. This can be wasteful if things like RVs are outside the window, but it's easy to reason about at the moment and it has the effect of simply throttling the speed for the watcher or lister.

This retries only certain mutating request errors. The ones that I know mean "this action didn't do anything".

[deads2k]

/retest

[hexfusion]

/retest

[deads2k]

/refresh

[deads2k]

/retest

[hexfusion]

/test gcp-upgrade

[openshift-ci-robot]

@hexfusion: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

• /test artifacts
• /test e2e-aws
• /test e2e-aws-csi
• /test e2e-aws-disruptive
• /test e2e-aws-fips
• /test e2e-aws-jenkins
• /test e2e-aws-multitenant
• /test e2e-aws-ovn
• /test e2e-aws-serial
• /test e2e-azure
• /test e2e-cmd
• /test e2e-gcp
• /test e2e-gcp-upgrade
• /test e2e-vsphere
• /test images
• /test integration
• /test k8s-e2e-conformance-aws
• /test k8s-e2e-gcp
• /test unit
• /test verify
• /test verify-commits

Use /test all to run the following jobs:

• pull-ci-openshift-kubernetes-master-e2e-aws-csi
• pull-ci-openshift-kubernetes-master-e2e-aws-fips
• pull-ci-openshift-kubernetes-master-e2e-aws-serial
• pull-ci-openshift-kubernetes-master-e2e-cmd
• pull-ci-openshift-kubernetes-master-e2e-gcp
• pull-ci-openshift-kubernetes-master-e2e-gcp-upgrade
• pull-ci-openshift-kubernetes-master-images
• pull-ci-openshift-kubernetes-master-integration
• pull-ci-openshift-kubernetes-master-k8s-e2e-gcp
• pull-ci-openshift-kubernetes-master-unit
• pull-ci-openshift-kubernetes-master-verify
• pull-ci-openshift-kubernetes-master-verify-commits

In response to this:

/test gcp-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hexfusion]

/test e2e-gcp-upgrade

[hexfusion]

/test e2e-gcp-upgrade

[deads2k]

/retest

[hexfusion]

need more data

/test e2e-gcp-upgrade

[hexfusion]

cluster-bot minor upgrade

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/1300830597689643008

[hexfusion]

minor upgrade #2
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/1300833926452875264

[tkashem]

/retest

[hexfusion]

FTR both upgrade jobs passed

#322 (comment)

#322 (comment)

[tkashem]

taking over the work, moving it to #327

[tkashem]

I think I have an understanding of what's causing some of the tests to fail - rbac bootstrapping takes longer than the test timeout
I0901 23:26:27.121787 21260 healthz.go:239] healthz check failed: poststarthook/rbac/bootstrap-roles

testserver.go:264: failed to launch server: failed to wait for /healthz to return ok: timed out waiting for the condition

This is the wait function in testserver.go:

	// wait until healthz endpoint returns ok
	err = wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) {
		select {
		case err := <-errCh:
			return false, err
		default:
		}

		result := client.CoreV1().RESTClient().Get().AbsPath("/healthz").Do(context.TODO())
		status := 0
		result.StatusCode(&status)
		if status == 200 {
			return true, nil
		}
		return false, nil
	})
	if err != nil {
		return result, fmt.Errorf("failed to wait for /healthz to return ok: %v", err)
	}

I am experimenting with aggressively reducing the retry backoff delay here #327. The integration job passes with much smaller retry delay https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_kubernetes/327/pull-ci-openshift-kubernetes-master-integration/1301014246380802048

So the next steps are:

• Investigate why rbac bootstrapping takes longer.
• Increase the test health endpoint timeout or use much shorter delay (need to figure out which is the best course of action)
• wire the context into the backed off retry delay so that the wait never surpasses the request context deadline
• ensure the delay is jittered to avoid cluster of retries.

[deads2k]

• Investigate why rbac bootstrapping takes longer.
• Increase the test health endpoint timeout or use much shorter delay (need to figure out which is the best course of action)
• wire the context into the backed off retry delay so that the wait never surpasses the request context deadline
• ensure the delay is jittered to avoid cluster of retries.

Think ugly. We can do this by having global to turn it off and on.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kube-apiserver/OWNERS [deads2k]
• staging/src/k8s.io/apiserver/OWNERS [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[deads2k]

/retest

[openshift-ci-robot]

@deads2k: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/verify	2e86cf6	link	/test verify
ci/prow/e2e-aws-fips	2e86cf6	link	/test e2e-aws-fips
ci/prow/e2e-gcp	2e86cf6	link	/test e2e-gcp
ci/prow/e2e-gcp-upgrade	2e86cf6	link	/test e2e-gcp-upgrade
ci/prow/k8s-e2e-gcp	2e86cf6	link	/test k8s-e2e-gcp
ci/prow/e2e-cmd	2e86cf6	link	/test e2e-cmd
ci/prow/e2e-aws-csi	2e86cf6	link	/test e2e-aws-csi

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[tkashem]

closing in favor of #327

/close

[openshift-ci-robot]

@tkashem: Closed this PR.

In response to this:

closing in favor of #327

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.