deflake kube-root-ca.crt e2e test

[zshihang]

What type of PR is this?

/kind flake

What this PR does / why we need it:

flaky in https://k8s-testgrid.appspot.com/sig-auth-gce#gce

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[zshihang]

/cc @liggitt

/sig auth
/priority important-soon
/triage accepted

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zshihang
To complete the pull request process, please assign tallclair after the PR has been reviewed.
You can assign the PR to them by writing /assign @tallclair in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• test/e2e/auth/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

The controller is logging many attempt to create that are getting 404 errors:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"9bbb8140-e6bd-4912-857d-031edc9e1cbe","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/persistent-local-volumes-test-6923/configmaps","verb":"create","user":{"username":"system:serviceaccount:kube-system:root-ca-cert-publisher","uid":"64d531b6-6fc8-4255-9b0a-069317c8a0d0","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"sourceIPs":["::1"],"userAgent":"kube-controller-manager/v1.20.0 (linux/amd64) kubernetes/ac62c47/system:serviceaccount:kube-system:root-ca-cert-publisher","objectRef":{"resource":"configmaps","namespace":"persistent-local-volumes-test-6923","name":"kube-root-ca.crt","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2020-11-05T16:29:16.609213Z","stageTimestamp":"2020-11-05T16:29:16.663037Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "system:controller:root-ca-cert-publisher" of ClusterRole "system:controller:root-ca-cert-publisher" to ServiceAccount "root-ca-cert-publisher/kube-system""}}

It is not handling the case where a namespace is deleted properly, which is backing up the queue with retry attempts. I think we need this:

diff --git a/pkg/controller/certificates/rootcacertpublisher/publisher.go b/pkg/controller/certificates/rootcacertpublisher/publisher.go
index 34fd3127a5d..ee8925cb3f9 100644
--- a/pkg/controller/certificates/rootcacertpublisher/publisher.go
+++ b/pkg/controller/certificates/rootcacertpublisher/publisher.go
@@ -187,6 +187,10 @@ func (c *Publisher) syncNamespace(ns string) error {
 				"ca.crt": string(c.rootCA),
 			},
 		}, metav1.CreateOptions{})
+		// don't retry a create if the namespace doesn't exist or is terminating
+		if apierrors.IsNotFound(err) || apierrors.HasStatusCause(err, v1.NamespaceTerminatingCause) {
+			return nil
+		}
 		return err
 	case err != nil:
 		return err

[liggitt]

I don't think this will be needed after #96277, but keep an eye on the testgrid to be sure

[zshihang]

passing after #96277