[flaky-test] Certificate Rotation

[bigkevmcd]

Flaky Test

=== RUN   Test_Operation_SetA_Custom_CertificateRotation
time="2024-05-22T13:56:19Z" level=info msg="Running in single server mode, will not peer connections"
    certificaterotation.go:43: cluster test-custom-certificate-rotation-operations rotate certificates wait failed on: rotate certificates wait did not succeed : timeout waiting condition: context deadline exceeded
        cluster test-custom-certificate-rotation-operations test data bundle:
--- FAIL: Test_Operation_SetA_Custom_CertificateRotation (1201.38s)

rancher/tests/v2prov/tests/custom/operation_certificaterotation_test.go

Line 15 in ee89ad3

func Test_Operation_SetA_Custom_CertificateRotation(t *testing.T) {

Release Branch
source branch of PR: release/v2.8

Drone Build (if applicable)

pipeline stage URL:

https://drone-pr.rancher.io/rancher/rancher/39110/4/2

[andreas-kupries]

While I am not seeing the flakiness of the cert rotation I see a more general breakage reported around cluster provisioning, it seems.

This is for a PR sitting on release/v2.9: #45269

In case it matters, a local k3s-based Rancher starts up just fine for that PR.

My latest drone logs are at

• https://drone-pr.rancher.io/rancher/rancher/39161
• https://drone-pr.rancher.io/rancher/rancher/39153

The message is generally the same across various failures:

... failed on: prov cluster is not ready: timeout waiting condition: context deadline exceeded

Question: Is each test creating its own cluster ? And removing it later ?
Because I see that there are passing tests too.

I see only 2 tests fail in in each of the 5 provisioning stages, mostly different across the stages.

Failing: Test_Provisioning_Custom_OneNodeWithDelete, Test_Provisioning_MP_SingleNodeAllRolesWithDelete, Test_Provisioning_Custom_ThreeNode, Test_Operation_SetA_Custom_CertificateRotation, Test_Operation_SetA_MP_CertificateRotation, Test_Operation_SetB_Custom_EtcdSnapshotOperationsOnNewCombinedNode, Test_Operation_SetB_MP_EtcdSnapshotOperationsWithThreeEtcdNodesOnNewNode

In the build-pr failures I see the same failed on ... message, after the unit tests were run and passed.

[andreas-kupries]

Created a PR without material code changes (comment fix).
Seeing failed builds there too, see https://drone-pr.rancher.io/rancher/rancher/39168
However none of the context deadline exceeded from my branch :(
Now wondering if the addition of the status field, and its handling slowed something down enough to trigger these timeouts.

[bigkevmcd]

Yeah, I'm still getting Test_Operation_SetB_MP_EtcdSnapshotOperationsWithThreeEtcdNodesOnNewNode failing.

For the nature of the change in this PR #45572, this should just not be affected.

[Oats87]

Doing investigation into this, I'm seeing that there were issues with operations taking significantly longer after v1.27.11+rke2r1 was released. It was almost a 200 second difference in my benchmark setup.

As such, as a temporary workaround, we can pin the RKE2 version to v1.27.10+rke2r1 for now, which should hopefully unblock CI.

[snasovich]

Adding to a milestone and some additional labels to ensure we circle back on this and address version pinning.

[slickwarren]

no QA required - closing this issue

[snasovich]

Reopening to track unpinning per #45577 (comment). Changing milestone so it doesn't appear it's blocking 2.9.0 release.