Improve certificate condition checking and error logging #4298

inteon · 2021-08-03T17:52:53Z

What this PR does / why we need it:

In #4234 there were intermittent test failures in the cainjector E2E tests:

https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/jetstack_cert-manager/4234/pull-cert-manager-e2e-v1-21/1422524737686343681
https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/jetstack_cert-manager/4234/pull-cert-manager-e2e-v1-21/1421109702468571136

In #4239, these flakes were partially resolved, although there is still an uncovered edge case where the flakes still persist.
This PR introduces more robust checking for both conditions Ready=true and Issuing not set.
Additionally it improves the logging in case of an error.
The logic for doing this is now aggregated in test/e2e/framework/helper/certificates.go.
The certificate functions in test/e2e/util/util.go are now not used anymore and deprecation comments have been added.

Release note:

/kind flake

jetstack-bot · 2021-08-03T17:52:55Z

@inteon: The label(s) kind/bugfix cannot be applied, because the repository doesn't have them.

In response to this:

What this PR does / why we need it:

In #4234 there were intermittent test failures in the cainjector E2E tests:

https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/jetstack_cert-manager/4234/pull-cert-manager-e2e-v1-21/1422524737686343681
https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/jetstack_cert-manager/4234/pull-cert-manager-e2e-v1-21/1421109702468571136

In #4239, these flakes were partially resolved, although there is still an uncovered edge case where the flakes still persist.
This PR introduces more robust checking for both conditions Ready=true and Issuing not set.
Additionally it improves the logging in case of an error.
The logic for doing this is now aggregated in test/e2e/framework/helper/certificates.go.
The certificate functions in test/e2e/util/util.go are now not used anymore and deprecation comments have been added.

Release note:
/kind bugfix

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jetstack-bot · 2021-08-03T17:53:08Z

Hi @inteon. Thanks for your PR.

I'm waiting for a jetstack or cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

irbekrm · 2021-08-03T17:57:14Z

/ok-to-test

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

wallrj

Thanks @inteon

I'm struggling to get my head round the changes and I've asked a bunch of (potentially stupid) questions below.

Please expand the comments for the helper functions so that it's clear to users like me how they work and when they should be used.

Also please write a bit more about exactly what is wrong with the existing code that causes the tests to still be flaky.

test/e2e/framework/helper/certificates.go

test/e2e/util/util.go

wallrj · 2021-08-04T11:54:14Z

/kind cleanup

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

wallrj

Thanks @inteon

There's one thing that looks like a mistake (waiting for Certificate with name of Secret)
But all my other comments are optional.

Please address the points or answer them and unhold this if you think it's ok to merge.

/hold
/lgtm

test/e2e/framework/helper/certificates.go

test/e2e/suite/conformance/certificates/tests.go

test/e2e/suite/issuers/acme/certificate/http01.go

test/e2e/suite/issuers/ca/certificate.go

wallrj · 2021-08-04T16:34:02Z

test/e2e/suite/serving/cainjector.go


 				By("changing the name of the corresponding secret in the cert")
-				secretName := types.NamespacedName{Name: cert.Spec.SecretName, Namespace: f.Namespace.Name}
+				cert = cert.DeepCopy() // DeepCopy before updating


Ah, was that the cause of the intermittent failure?
Did you consider doing the DeepCopy in the WaitForCertificate... helper functions, to help avoid this same bug happening elsewhere?

This could be related, but I fear that there are multiple errors causing failures.
I would like to keep the cert = cert.DeepCopy() visible in the code, so that we can more easily see when DeepCopies are needed.

test/e2e/util/util.go

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

wallrj

Thanks for the latest adjustments @inteon

/lgtm
/approve
/kind cleanup

wallrj · 2021-08-05T08:16:31Z

test/e2e/suite/issuers/acme/certificate/http01.go

 		By("Waiting for Certificate to exist")
-		err := util.WaitForCertificateToExist(certClient, certificateSecretName, foreverTestTimeout)
+		cert, err := f.Helper().WaitForCertificateToExist(f.Namespace.Name, certificateSecretName, time.Second*60)


Here again, we seem to be using the name of the Secret as the name of the Certificate.

Suggested change

cert, err := f.Helper().WaitForCertificateToExist(f.Namespace.Name, certificateSecretName, time.Second*60)

cert, err := f.Helper().WaitForCertificateToExist(f.Namespace.Name, certificateName, time.Second*60)

I guess it works because both names are the same.

I can't face another round of code review and E2E tests. Let's just leave it.

jetstack-bot · 2021-08-05T08:18:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon, wallrj

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [wallrj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wallrj · 2021-08-07T15:41:57Z

/cherry-pick release-1.5

jetstack-bot · 2021-08-07T15:42:32Z

@wallrj: new pull request created: #4322

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jetstack-bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 3, 2021

inteon force-pushed the fix_test_flake branch 3 times, most recently from 180815f to 6dd0c94 Compare August 3, 2021 19:03

improved certificate condition checking and error logging

2d2bde5

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

inteon force-pushed the fix_test_flake branch from 6dd0c94 to 2d2bde5 Compare August 3, 2021 21:00

inteon mentioned this pull request Aug 3, 2021

Cleanup tests #4300

Merged

wallrj requested changes Aug 4, 2021

View reviewed changes

jetstack-bot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Aug 4, 2021

cleanup test functions

7bf6bf9

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

jetstack-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 4, 2021

inteon force-pushed the fix_test_flake branch 2 times, most recently from 4d27e2c to e388d68 Compare August 4, 2021 15:08

wallrj approved these changes Aug 4, 2021

View reviewed changes

jetstack-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 4, 2021

jetstack-bot assigned wallrj Aug 4, 2021

jetstack-bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Aug 4, 2021

wallrj mentioned this pull request Aug 4, 2021

Fix a cainjector E2E test flake #4294

Closed

inteon force-pushed the fix_test_flake branch from e388d68 to fa95eca Compare August 4, 2021 16:43

jetstack-bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 4, 2021

inteon force-pushed the fix_test_flake branch from fa95eca to 4a46c48 Compare August 4, 2021 16:49

resolve bugs

66820ed

Signed-off-by: Inteon <42113979+inteon@users.noreply.github.com>

inteon force-pushed the fix_test_flake branch from 4a46c48 to 66820ed Compare August 4, 2021 17:26

inteon requested a review from wallrj August 4, 2021 19:03

wallrj approved these changes Aug 5, 2021

View reviewed changes

jetstack-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 5, 2021

jetstack-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 5, 2021

jetstack-bot merged commit 8d0c228 into cert-manager:master Aug 5, 2021

jetstack-bot added this to the v1.5 milestone Aug 5, 2021

jetstack-bot mentioned this pull request Aug 7, 2021

[release-1.5] Improve certificate condition checking and error logging #4322

Merged

inteon deleted the fix_test_flake branch August 10, 2021 09:49

This was referenced Mar 6, 2022

Retry on conflict for the end-to-end test "added an additional dnsName" #4924

Merged

Retry on conflict for the end-to-end test "CA Injector for api services should update data when the certificate changes" #4925

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve certificate condition checking and error logging #4298

Improve certificate condition checking and error logging #4298

inteon commented Aug 3, 2021 •

edited

Loading

jetstack-bot commented Aug 3, 2021

jetstack-bot commented Aug 3, 2021

irbekrm commented Aug 3, 2021

wallrj left a comment

wallrj commented Aug 4, 2021

wallrj left a comment

wallrj Aug 4, 2021

inteon Aug 4, 2021

wallrj left a comment

wallrj Aug 5, 2021

jetstack-bot commented Aug 5, 2021

wallrj commented Aug 7, 2021

jetstack-bot commented Aug 7, 2021

	cert, err := f.Helper().WaitForCertificateToExist(f.Namespace.Name, certificateSecretName, time.Second*60)
	cert, err := f.Helper().WaitForCertificateToExist(f.Namespace.Name, certificateName, time.Second*60)

Improve certificate condition checking and error logging #4298

Improve certificate condition checking and error logging #4298

Conversation

inteon commented Aug 3, 2021 • edited Loading

jetstack-bot commented Aug 3, 2021

jetstack-bot commented Aug 3, 2021

irbekrm commented Aug 3, 2021

wallrj left a comment

Choose a reason for hiding this comment

wallrj commented Aug 4, 2021

wallrj left a comment

Choose a reason for hiding this comment

wallrj Aug 4, 2021

Choose a reason for hiding this comment

inteon Aug 4, 2021

Choose a reason for hiding this comment

wallrj left a comment

Choose a reason for hiding this comment

wallrj Aug 5, 2021

Choose a reason for hiding this comment

jetstack-bot commented Aug 5, 2021

wallrj commented Aug 7, 2021

jetstack-bot commented Aug 7, 2021

inteon commented Aug 3, 2021 •

edited

Loading