Let's Encrypt: Unable to get staging certificates

[mitch-karch]

I'm having trouble getting a certificate from the staging step, here is the log:

---- Log start at 12/2/2019, 9:10:17 PM ----
9:10:17 PM - main: Core module windows-amt is 368913 bytes.
9:10:17 PM - main: Core module linux-amt is 340820 bytes.
9:10:17 PM - main: Core module linux-noamt is 248457 bytes.
9:10:17 PM - main: Core module windows-recovery is 44289 bytes.
9:10:17 PM - main: Core module linux-recovery is 26030 bytes.
9:10:17 PM - main: Core module windows-agentrecovery is 24624 bytes.
9:10:17 PM - main: Core module linux-agentrecovery is 6365 bytes.
9:10:17 PM - cert: Initializing Let's Encrypt support, using GreenLock v3.1.5
9:10:17 PM - cert: Getting certs from local store
9:10:18 PM - cert: Notify: error: {"length":0}
9:10:18 PM - cert: Unable to get certs from local store (Staging)
9:10:19 PM - main: Server started
9:10:28 PM - cert: Checking certificate for meshcentral.xxxxxxxx.com (Staging)
9:10:29 PM - cert: Notify: error: {"length":0}
9:10:29 PM - cert: Unable to get a certificate (Staging, 1003ms): [{"site":{"subject":"meshcentral.xxxxxxxx.com","altnames":["meshcentral.xxxxxxxx.com"]},
"error":{"0":"e","1":"r","2":"r","3":"o","4":"r","length":5}}]

My Config has the following:

  "letsencrypt": {
    "email": "xxxxxx@xxxxxx.com ",
    "names": "meshcentral.xxxxxx.com",
    "rsaKeySize": 2048,
    "production": false
  },

Currently running:

• Node v12.13.1
• Meshcentral 0.4.5-c

Edit: letsdebug shows "All OK!"

[ryanblenis]

I'm sure Ylian will have a better response, but you may want to include what https://letsdebug.net/ has to say about the site as well for more info.

[mitch-karch]

Yes, forgot to add that. Letsdebug shows no errors

[Ylianst]

Interesting, you did everything correctly here. I was exactly going to suggest https://letsdebug.net/, so very nice that you did that. By the way, "rsaKeySize" is now ignored and always 2048 in this version, so that has no impact. Your settings look good. Even if you used "production":true, this new MeshCentral will always first start by getting a staging certificate. So, you need to pass that test anyway regardless. It looks like it's failing after almost exactly 1 second.

First, make sure the "meshcentral-data" folder is writable. GreenLock3 will need to create and write new files to that folder. If you already have "letencrypt3" and "letencrypt3-staging" folders in meshcentral-data, remove them and try again. These folders should be auto-recreated.

One thing you could try is running MeshCentral like this:

node node_modules/meshcentral --debug cert,web,webrequest

You will now see all the web requests coming into the server on both ports 80 and 443. Then, you can try https://letsdebug.net/ again and make sure that you see the request from the letsdebug server or get requests from the Let's Encrypt staging server. If by chance, let's debug and let's encrypt try to access your server on port 80 and hit a different server, Let's Debug may say ok, but Let's Encrypt will not like that.

Are you running a reverse proxy in front of MeshCentral? If you can try the two things above and report back, that would be great. - Thanks.

[mitch-karch]

As far as I can tell, the folders should be writable, is there something about the specific folder permissions maybe? I'm noticing that the log.txt file in meshcentral-data is being generated without read permission, i'm not sure if that's a clue or a red herring. The folders for letsencrypt have "drwxrwxr-x"

The system is running on a fresh Azure VM

Here is the log file with the extra debug flags as well as trying letsdebug-test (still reporting "OK")

$ node node_modules/meshcentral/ --debug cert,web,webrequest
MeshCentral HTTP redirection server running on port 80.
CERT: Initializing Let's Encrypt support, using GreenLock v3.1.5
CERT: Getting certs from local store
ACME Directory URL: https://acme-staging-v02.api.letsencrypt.org/directory
CERT: Notify: error: {"length":0}
CERT: Unable to get certs from local store (Staging)
MeshCentral v0.4.5-c, WAN mode.
MeshCentral Intel(R) AMT server running on meshcentral.xxxxx.com:4433.
MeshCentral HTTPS server running on meshcentral.xxxxx.com:443.
WEBREQUEST: /agent.ashx/.websocket
WEBREQUEST: /agent.ashx/.websocket
CERT: Checking certificate for meshcentral.xxxxx.com (Staging)
CERT: Notify: error: {"length":0}
CERT: Unable to get a certificate (Staging, 1004ms): [{"site":{"subject":"meshcentral.xxxxx.com","altnames":["meshcentral.xxxxx.com"]},"error":{"0":"e","1":"r","2":"r","3":"o","4":"r","length":5}}]
WEBREQUEST: /.well-known/acme-challenge/letsdebug-test (RedirServer)
CERT: Challenge meshcentral.xxxxx.com/letsdebug-test
WEBREQUEST: / (RedirServer)
WEBREQUEST: /.well-known/acme-challenge/yyyyyy(RedirServer)
CERT: Challenge meshcentral.xxxxx.com/yyyyyy
WEBREQUEST: /.well-known/acme-challenge/yyyyyy(RedirServer)
CERT: Challenge meshcentral.xxxxx.com/yyyyyy
WEBREQUEST: /.well-known/acme-challenge/yyyyyy(RedirServer)
CERT: Challenge meshcentral.xxxxx.com/yyyyyy
WEBREQUEST: /.well-known/acme-challenge/yyyyyy(RedirServer)
CERT: Challenge meshcentral.xxxxx.com/yyyyyy

[Ylianst]

This is interesting. It's as if GreenLock3 is timing out after exactly 1 second, but you are getting the challenge requests from the Let's Encrypt server perfectly... Thanks for posting the logs, let me work on this a bit.

[Ylianst]

Just posted an issue on GreenLock here on this topic because I would like better error messages from GreenLock.

[Ylianst]

Looking at the GreenLock3 code, I did find that it seems like the domain of your email address will get DNS resolved and there is a exact 1 second timeout for that. So, it could be that your email address domain name does not resolve and causes the fail. Can you revise your email address and try again? - If that is not the problem, let me know.

[Ylianst]

Just published MeshCentral v0.4.5-f that will now display the error strings when running manually. So, it's possible that with this new version, a GreenLock3 problem will now show up.

[mitch-karch]

Ah ha! A single trailing whitespace character in the config.js for my email. Certificate issues just fine now. Thanks for all your help!