Unreliable with load balancers and API Gateways

[mattock]

We're using certbot-s3front to manage certificates for Amazon Load Balancers (ELBv2) and API Gateways. The problem is that renewal fails very often - possibly 50% of the time - but when renewal is retried a few times manually it usually succeeds. So the setup is not completely broken but renewal fails way too often than it should. In the logs we typically get an "Internal server error":

2020-03-01 08:56:06,588:DEBUG:urllib3.connectionpool:https://stats.example.org:443 "GET /.well-known/acme-challenge/MsxpjEhYSbZ5BNZllTKrS6QtjvHRmfsKbixgYZRxFrs HTTP/1.1" 502 3
6
2020-03-01 08:56:06,589:DEBUG:acme.challenges:Received <Response [502]>: {"message": "Internal server error"}. Headers: {'Content-Type': 'application/json', 'Content-Length': '36', 'Connection': 'keep-alive', 'Date': 'Sun, 01 Mar 2020 08:56:06 GMT', 'x-amzn-RequestId': '1af03733-c639-4ad2-914a-a6d02c6b8ea2', 'x-amzn-ErrorType': 'InternalServerErrorException', 'x-amz-apigw-id': 'Is_Jkhn5yK7Fp1c=', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 c4a9b650ba57ac8421645370927ef0cf.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'SFO20-C1', 'X-Amz-Cf-Id': 'OHm2S8od_qvjDO_UuS__sy34duibiArzjN_LA92hVtHHtRtc4lGtKg=='}

Or we may get a "502 Bad Gateway":

  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://api.example.org/.well-known/acme-challenge/yIuGO1v8PvyHbrCPMNXnFzB0ZMzv0mC22QBFi3E52gk [a.b.c.d]: \"\u003chtml\u003e\\r\\n\u003chead\u003e\u003ctitle\u003e502 Bad Gateway\u003c/title\u003e\u003c/head\u003e\\r\\n\u003cbody bgcolor=\\\"white\\\"\u003e\\r\\n\u003ccenter\u003e\u003ch1\u003e502 Bad Gateway\u003c/h1\u003e\u003c/center\u003e\\r\\n\u003c/body\u003e\\r\"",
        "status": 403
      },

These cause self-verification to fail:

2020-03-01 08:56:06,590:ERROR:certbot_s3front.authenticator:Self-verify of challenge failed, authorization abandoned!

The only thing I can think of is that sometimes the verification file is placed to and searched from a different place, which makes renewal work sometimes, but not consistently. As in "load balancer forwards traffic to the wrong place".

We use the following software, installed using pip3 on Ubuntu 18.04:

• certbot (0.34.2)
• certbot-s3front (0.4.2)

Is this is a known problem and is there something we can do about it?

[dlapiduz]

@mattock to be honest, this is the first I hear about it... do you have the load balancer and cloud front in line? Maybe you can try configuring the cloudfront endpoint to point directly to S3?