The cert-manager version we are using is not supported by letsencrypt.

[cjwagner]

We've been working with Jetstack, the authors of cert-manager, on a
series of fixes to the client. Cert-manager sometimes falls into a
traffic pattern where it sends really excessive traffic to Let's
Encrypt's servers, continuously. To mitigate this, we plan to start
blocking all traffic from cert-manager versions less than 0.8.0 (the
current semver minor release), as of November 1, 2019. Please upgrade
all of your cert-manager instances before then.

We're sending this email because this is the contact address of your
cert-manager instance at:

35.185.195.73 .

Version 0.8.0 is much better but we still observe excessive traffic in
some cases. We're working with Jetstack to improve these cases. As new
versions of cert-manager are released, we will add the non-current
versions to our block list after 3 months. We strongly encourage
cert-manager users to stay up-to-date with new versions.

You can subscribe to new version notifications by going to cert-manager's
GitHub project (https://github.com/jetstack/cert-manager), clicking
"Watch" in the upper right, then clicking "Releases only."

Also, there is an opportunity to help both Jetstack and Let's Encrypt.
Once you've upgraded, please check the logs for your cert-manager
instances from time to time. Are they making excessive requests to Let's
Encrypt (more than, say, 10 per day over multiple days)? If so, please
share details at cert-manager/cert-manager#1948 .

Thanks,
Let's Encrypt Team

We are currently using version v0.5.2:

test-infra/prow/cluster/cert-manager.yaml

Line 155 in 7abdd3f

image: "quay.io/jetstack/cert-manager-controller:v0.5.2"

Unfortunately there are backwards incompatible changes that make upgrading cert-manager more complicated than just changing the image version: https://docs.cert-manager.io/en/latest/tasks/upgrading/

The current certificate for prow.k8s.io is good until Monday, January 6, 2020 at 11:47:17 AM

/kind oncall-hotlist
@Katharine @fejta @BenTheElder @stevekuznetsov

I know we have discussed using Google managed SSL certificates for prow.k8s.io. If we want to make that change now would be a good time.

[Katharine]

I am very inclined to switch to GKE-managed certs.

[Katharine]

see #14945 for what that would ultimately look like.

[BenTheElder]

what's the current status on this?

[Katharine]

We need someone (@fejta, probably) to change over the kubernetes and kubernetes-sigs webhooks to point at https://prow-canary.k8s.io instead of https://prow.k8s.io, and then we can perform the switchover during some low-traffic time.

We have until January 6th to actually execute this plan, but sooner is better than later.

[fejta]

Switching https://prow.k8s.io/hook to https://prow-canary.k8s.io/

[fejta]

Done

[Katharine]

Switching https://prow.k8s.io/hook to https://prow-canary.k8s.io/

Do you mean https://prow-canary.k8s.io/hook?

[fejta]

Yes, and we're getting 200s

Request URL: https://prow-canary.k8s.io/hook
Request method: POST
content-type: application/json
Expect: 
User-Agent: GitHub-Hookshot/f1003bc
X-GitHub-Delivery: bffba280-01b1-11ea-95ad-0e33a0477690
X-GitHub-Event: issue_comment
X-Hub-Signature: sha1=bda9df71ef463adf5afc1829fa3c08b1ec25ff64

[Katharine]

/pony

[k8s-ci-robot]

@Katharine:

In response to this:

/pony

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta]

/assign @fejta @Katharine @clarketm

Want to try and flip this over this week?

[fejta]

Bueller?

[Katharine]

Fixed by #15726, #15727 and #15728.