Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix TestFileIntegrityCertRotation e2e test #440

Merged
merged 1 commit into from
Sep 27, 2023

Conversation

Vincent056
Copy link
Contributor

Change the how we check if a kubelet cert rotation is being completed, and make sure FIO object pass after the rotation

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 19, 2023
@rhmdnd
Copy link
Contributor

rhmdnd commented Sep 20, 2023

Is this related to either of the open issues we have for broken e2e tests?

#438
#439

@Vincent056
Copy link
Contributor Author

Is this related to either of the open issues we have for broken e2e tests?

#438 #439

yes it is, we need to change how we rotate the cert, the old way was not working anymore

@Vincent056 Vincent056 force-pushed the rotate_cert branch 2 times, most recently from 016cbe5 to 959d9fe Compare September 21, 2023 01:17
@rhmdnd
Copy link
Contributor

rhmdnd commented Sep 21, 2023

/retest

Looks like the e2e setup failed due to ImagePullBackoff (seemed unrelated).

@Vincent056
Copy link
Contributor Author

/retest

@Vincent056
Copy link
Contributor Author

failed due to ImagePullBackoff

@Vincent056
Copy link
Contributor Author

/retest

@Vincent056 Vincent056 force-pushed the rotate_cert branch 2 times, most recently from dd16976 to a9be543 Compare September 27, 2023 06:37
@Vincent056
Copy link
Contributor Author

TestFileIntegrityCertRotation is fixed, however there are issue with node taint on bundle e2e test

Change the how we check if a kubelet cert rotation is being completed, and make sure FIO object pass after the rotation
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 27, 2023

@Vincent056: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

if secret.Annotations == nil {
secret.Annotations = map[string]string{}
// wait for cert rotation daemonset to be ready
err = waitForDaemonSet(daemonSetIsReady(f.KubeClient, "kubelet-bootstrap-cred-manager", "openshift-machine-config-operator"))
Copy link
Contributor

@rhmdnd rhmdnd Sep 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Potential area for improvement (although I don't expect us to do it in this patch), would be to consider a simplified polling method here, instead of a nested one:

  name := "kubelet-bootstrap-cred-manager"
  namespace := "openshift-machine-config-operator"
  err = assertDaemonSetIsActive(name, namespace)

t.Errorf("Timed out waiting for DaemonSet kubelet-bootstrap-cred-manager")
}

// Delete the secrets csr-signer-signer and csr-signer from the openshift-kube-controller-manager-operator namespace
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do this because it's what forces the rotation, right?

}
taintedNode.Spec.Taints = append(taintedNode.Spec.Taints, taint)
t.Logf("Tainting node: %s", taintedNode.Name)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting - so what this causing issues in previous CI runs because it wasn't being retried? Or we're working around the CI issue because we're retrying?

@@ -1755,38 +2030,45 @@ func waitUntilPodsAreGone(t *testing.T, c client.Client, pods *corev1.PodList, i
}

func taintNode(t *testing.T, f *framework.Framework, node *corev1.Node, taint corev1.Taint) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Taints are specific to nodes, we could simplify the name to addTaint.

taintedNode.Spec.Taints = []corev1.Taint{}
}
taintedNode.Spec.Taints = append(taintedNode.Spec.Taints, taint)
t.Logf("Tainting node: %s", taintedNode.Name)
return f.Client.Update(goctx.TODO(), taintedNode)
},
)
}

func removeNodeTaint(t *testing.T, f *framework.Framework, nodeName, taintKey string) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Similar comment as above, we could simplify to removeTaint

return retryDefault(
func() error {
// taint with retry
// let's fetch the latest node object first
fetchedNode := &corev1.Node{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to fetch the node again, even though it's passed in, because we're not sure if it has been updated previously and cause our update to fail because we're attempting to update an older version?

Copy link
Contributor

@rhmdnd rhmdnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only have minor suggestions inline, most of which we can address in subsequent patches if we decide to do so.

Thanks for fixing up the CI, @Vincent056

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 27, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 27, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhmdnd, Vincent056

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 467e23a into openshift:master Sep 27, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants