Bug 1841013: TestBoundTokenSignerController tests is unstable causing e2e-aws-operator to fail frequently #868

p0lyn0mial · 2020-05-27T08:46:12Z

No description provided.

sttts · 2020-05-27T08:51:38Z

test/e2e/bound_sa_token_test.go

 	err := wait.PollImmediate(interval, timeout, func() (done bool, err error) {
+


nit: no empty line

sttts · 2020-05-27T08:52:31Z

test/e2e/bound_sa_token_test.go

 	})
 }

 // checkBoundTokenOperandSecret checks that the operand secret is
 // populated with the expected data.
-func checkBoundTokenOperandSecret(t *testing.T, kubeClient *clientcorev1.CoreV1Client, timeout time.Duration, expectedData map[string][]byte) {
+func checkBoundTokenOperandSecret(t *testing.T, kubeClient *clientcorev1.CoreV1Client, additionalCheckFn func(), timeout time.Duration, expectedData map[string][]byte) {


doc additionalCheckFn

sttts · 2020-05-27T08:53:26Z

Needs a PR description what is actually done.

p0lyn0mial · 2020-05-27T10:09:15Z

/test e2e-aws-operator

p0lyn0mial · 2020-05-27T11:46:07Z

/test e2e-aws-operator

p0lyn0mial · 2020-05-27T12:09:33Z

I think that the issue is that the test doesn't know how to detect that a new version of the server has been rolled out and it's okay to go.

Rolling out a new version is an expensive process that takes quite a long time (30 min?).
Likewise triggering a new release - it's not instant - I suspect it takes a few minutes between "I want I new version" and "OK, I'm starting rolling out a new version"

Like this particular line - I suspect it almost always yields immediately which will cause the test to think it's okay to test . . .

cluster-kube-apiserver-operator/test/e2e/bound_sa_token_test.go

Line 84 in 31f3997

    
           test.WaitForKubeAPIServerClusterOperatorAvailableNotProgressingNotDegraded(t, configClient)

I increased the timeout and added an additional check that will wait for the operator (AvailableNotProgressingNotDegraded) while waiting for the token and it seems to improve the test.

p0lyn0mial · 2020-05-27T12:12:40Z

I think that we could change WaitForKubeAPIServerClusterOperatorAvailableNotProgressingNotDegraded function.

It could read revisions and wait until all replicas are on the same revision. It could then report OK only when the same revision has been seen for X minutes

p0lyn0mial · 2020-05-27T12:16:11Z

I think that increasing only the timeout will fix the issue - trying now.

p0lyn0mial · 2020-05-27T12:45:30Z

I would also remove the usage of WaitForKubeAPIServerClusterOperatorAvailableNotProgressingNotDegradedfrom:

p0lyn0mial · 2020-05-27T12:50:42Z

I think it also has a few more issues, for example, TestEncryptionTypeAESCBC can start rolling out a new version and fail before completion. That will definitely impact the next tests - as they expect a stable environment and will very likely fail

p0lyn0mial · 2020-05-27T14:01:15Z

/assign @marun

p0lyn0mial · 2020-05-27T14:20:59Z

/retest

marun · 2020-05-27T14:51:35Z

/lgtm

sttts · 2020-05-27T15:31:28Z

/approve

openshift-ci-robot · 2020-05-27T15:32:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: marun, p0lyn0mial, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-05-28T07:02:17Z

@p0lyn0mial: This pull request references Bugzilla bug 1841013, which is invalid:

expected the bug to target the "4.5.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1841013: TestBoundTokenSignerController tests is unstable causing e2e-aws-operator to fail frequently

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

p0lyn0mial · 2020-05-28T07:02:42Z

/bugzilla refresh

openshift-ci-robot · 2020-05-28T07:02:48Z

@p0lyn0mial: This pull request references Bugzilla bug 1841013, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2020-05-28T10:49:50Z

/retest