Unable to get Letsencrypt certificate #2498

chc-pr · 2019-04-04T20:56:33Z

I have followed the installation guide at https://mailcow.github.io/mailcow-dockerized-docs/i_u_m_install/
but am getting the same problem on fresh installs on both Ubuntu 18.04.2 ND 16.04.3, Letsencrypt fails to get a certificate.

I know this looks the same as other issues, but I have tried everything I could find so far without success.

Installs are on VMware 6.7U1 VMs set with 2 cores, and 4GB ram.
Physical host dual is Xeon E5-2630. Host loading is <5% max and often around 2%, 19GB is assigned in total representing 60% of the systems physical RAM - so I don't think it is loading related.

Running dig +short myip.opendns.com @resolver1.opendns.com returns correct public IP address. In addition I can access both http and https pages from an outside network (GSM network on my phone).

running ... /opt/mailcow-dockerized# docker-compose logs acme-mailcow
returns .... (this example is from 18.04, but its pretty identical in both systems) ...

WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
Attaching to mailcowdockerized_acme-mailcow_1
acme-mailcow_1 | Thu Apr 4 14:05:45 BST 2019 - Waiting for Docker API...OK
acme-mailcow_1 | Thu Apr 4 14:05:45 BST 2019 - Waiting for database... Uptime: 2 Threads: 9 Questions: 144 Slow queries: 0 Opens: 37 Flush tables: 1 Open tables: 27 Queries per second avg: 72.000
acme-mailcow_1 | OK
acme-mailcow_1 | Thu Apr 4 14:05:53 BST 2019 - Waiting for Nginx... OK
acme-mailcow_1 | Thu Apr 4 14:05:53 BST 2019 - Waiting for domain table... OK
acme-mailcow_1 | Thu Apr 4 14:06:03 BST 2019 - Initializing, please wait...
acme-mailcow_1 | Thu Apr 4 14:06:03 BST 2019 - Generating missing domain private key...
acme-mailcow_1 | Generating RSA private key, 4096 bit long modulus (2 primes)
acme-mailcow_1 | .......++++
acme-mailcow_1 | ..................................................................................++++
acme-mailcow_1 | e is 65537 (0x010001)
acme-mailcow_1 | Thu Apr 4 14:06:04 BST 2019 - Generating missing Lets Encrypt account key...
acme-mailcow_1 | Generating RSA private key, 4096 bit long modulus (2 primes)
acme-mailcow_1 | ..........................................................................................................++++
acme-mailcow_1 | ...........++++
acme-mailcow_1 | e is 65537 (0x010001)
acme-mailcow_1 | Thu Apr 4 14:06:05 BST 2019 - Detecting IP addresses... OK
acme-mailcow_1 | Thu Apr 4 14:06:29 BST 2019 - Found A record for mxserver.mydomain.net: xxx.xxx.xxx.49
acme-mailcow_1 | Thu Apr 4 14:06:29 BST 2019 - Confirmed A record xxx.xxx.xxx.49, but HTTP validation failed
acme-mailcow_1 | Thu Apr 4 14:06:29 BST 2019 - Cannot validate hostnames, skipping Let's Encrypt for 1 hour.
acme-mailcow_1 | Thu Apr 4 14:06:29 BST 2019 - Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently.
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Waiting for Docker API...OK
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Waiting for database... Uptime: 445 Threads: 29 Questions: 2323 Slow queries: 0 Opens: 93 Flush tables: 1 Open tables: 50 Queries per second avg: 5.220
acme-mailcow_1 | OK
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Waiting for Nginx... OK
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Waiting for domain table... OK
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Initializing, please wait...
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Using existing domain key /var/lib/acme/acme/key.pem
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Detecting IP addresses... OK
acme-mailcow_1 | Thu Apr 4 14:13:43 BST 2019 - Found A record for mxserver.mydomain.net: xxx.xxx.xxx.49
acme-mailcow_1 | Thu Apr 4 14:13:43 BST 2019 - Confirmed A record xxx.xxx.xxx.49, but HTTP validation failed
acme-mailcow_1 | Thu Apr 4 14:13:43 BST 2019 - Cannot validate hostnames, skipping Let's Encrypt for 1 hour.
acme-mailcow_1 | Thu Apr 4 14:13:43 BST 2019 - Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently.
#=============================================================

Considering these two lines in particular
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Using existing domain key /var/lib/acme/acme/key.pem
acme-mailcow_1 | Thu Apr 4 14:13:23 BST 2019 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem

Trying to run cd /var/lib/acme/acme/ returns an error. Checking through I find that although /var/lib exists, there is no acme subdirectory let alone acme/acme, so obviously no .pem files. Are these in some sort of container which is not showing up with a cd? I am not really familiar with docker yet, so I am a complete noob in this regard.

I suspect I have now hit the ratelimit for a week, so will not be able to do much until late next week, but I would be grateful for any suggestions. I am looking to move my mailserver from iredmail to mailcow, but it is proving very frustrating trying to get this cert sorted.

Thank you in advance for any help which may come. If I have missed any relevant info please accept my apologies, as I said I'm a complete noob wrt docker.

andryyy · 2019-04-05T04:36:03Z

Hi,

This is very helpful for us:

cd /opt/mailcow-dockerized
source mailcow.conf
touch data/web/.well-known/acme-challenge/1
# For IPv4, should return 2xx or 3xx
docker-compose exec acme-mailcow curl -4 http://${MAILCOW_HOSTNAME}/.well-known/acme-challenge/1 --write-out %{http_code}
# For IPv6, should return 2xx or 3xx
docker-compose exec acme-mailcow curl -6 http://${MAILCOW_HOSTNAME}/.well-known/acme-challenge/1 --write-out %{http_code}

One could disable that check, but it only arises other problems.

The containers should be able to connect to themselves over the public IP.

pbaeumel · 2019-04-05T06:29:01Z

Dear andryyy,

the same problem occured on my Mailcow.

Loglines of ACME:
5.4.2019, 08:20:46 | Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently.
5.4.2019, 08:20:46 | Cannot validate hostnames, skipping Let's Encrypt for 1 hour.
5.4.2019, 08:20:46 | Confirmed A record 185.x.x.x, but HTTP validation failed

Output of IPv4-Test:
curl: (7) Failed to connect to mx.xxxxxxx.eu port 80: Host is unreachable

Output of IPv6-Test:
curl: (6) Could not resolve host: mx.xxxxxxx.eu

Best regards,
MacGyver

andryyy · 2019-04-05T06:58:18Z

Then you need to check and fix your firewall. See the other issues about that, we cannot do that in mailcow. You need to be able to connect to your public hostname from within the containers. Disable ufw or firewalld.

pbaeumel · 2019-04-05T17:06:09Z

@andryyy
firewalld has been active; disabled it; now getting value 200 from IPv4-Test.

Unfortunately ACME-Log still show another error:

5.4.2019, 18:59:55 | Cannot validate hostnames, skipping Let's Encrypt for 1 hour.
5.4.2019, 18:59:55 | Cannot match your IP against hostname mx.xxxxxxx.eu (185.x.x.x)
5.4.2019, 18:59:55 | Found A record for mx.xxxxxxx.eu: 185.x.x.x

chc-pr · 2019-04-05T17:30:09Z

This is the same as me. There was something in one on the replies I haven't ad time to check, but I wonder if there may be an issue I clocked. I am not in front of a terminal right now, but there was a reference to one of the dockerised components 'should be able to access the server' from the public ip. If I understand what was being said, this /might/ be my problem. The VM is behind a firewall NAT (pfSense) and uses a split DNS so any call to the public ip from within the server will fail as there is no NAT reflection set. Is my interpretation correct? On 5 April 2019, 18:06, MacGyver2018 wrote: @andryyy firewalld has been active; disabled it; now getting value 200 from IPv4-Test. Unfortunately ACME-Log still show another error: 5.4.2019, 18:59:55Cannot validate hostnames, skipping Let's Encrypt for 1 hour. 5.4.2019, 18:59:55Cannot match your IP against hostname mx.xxxxxxx.eu (185.x.x.x) 5.4.2019, 18:59:55Found A record for mx.xxxxxxx.eu: 185.x.x.x — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

andryyy · 2019-04-05T18:00:43Z

It just cannot match your ip against the name. Set SNAT or disable the IP check. :)

…

Am 05.04.2019 um 19:06 schrieb MacGyver2018 ***@***.***>: @andryyy firewalld has been active; disabled it; now getting value 200 from IPv4-Test. Unfortunately ACME-Log still show another error: 5.4.2019, 18:59:55 Cannot validate hostnames, skipping Let's Encrypt for 1 hour. 5.4.2019, 18:59:55 Cannot match your IP against hostname mx.xxxxxxx.eu (185.x.x.x) 5.4.2019, 18:59:55 Found A record for mx.xxxxxxx.eu: 185.x.x.x — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

chc-pr · 2019-04-06T14:55:19Z

I have now tried running ...
cd /opt/mailcow-dockerized
source mailcow.conf
touch data/web/.well-known/acme-challenge/1
docker-compose exec acme-mailcow curl -4 http://${MAILCOW_HOSTNAME}/.well-known/acme-challenge/1 --write-out %{http_code}

and obtained ...
WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.

<title>404 Not Found</title>

404 Not Found

nginx 404root@mx4:/opt/mailcow-dockerized#

I am quite happy to do another fresh install as nothing is working yet, which is the recommended distro to build on, because I am clearly making no progress with Ubuntu. Would Debian be a better bet?

andryyy · 2019-04-06T18:51:04Z

It is freshly broken. It doesn't matter if you reinstall 200 times, when it is a non-working base to start with.

It works fine on Ubuntu. If you get a 404 you are doing something wrong. The curl request is not pointing to mailcows Nginx then.

chc-pr · 2019-04-06T19:32:47Z

Edited thought ...
Should I have pre-installed Nginx? I have understood that the install included this, especially as I can get the http and https sites showing up, but I thought I had better check ...

Original post follows ...
Hmmm, I followed this guide to the letter ...
https://mailcow.github.io/mailcow-dockerized-docs/prerequisite-system/ followed by https://mailcow.github.io/mailcow-dockerized-docs/i_u_m_install/ and then went through
https://mailcow.github.io/mailcow-dockerized-docs/firststeps-ssl/ which I believe are the official install guides.

When I say to the letter, I mean I literally SSH'd using Putty to the server (fresh install of Ubuntu 18.04.2 from the official repository), ran sudo -i to become root then cut and pasted the lines in turn all the way through. Can't have done any typos I can see, and I did the same thing with the Ubuntu 16.04.3 install.

Any ideas?

It is freshly broken. It doesn't matter if you reinstall 200 times, when it is a non-working base to start with.

It works fine on Ubuntu. If you get a 404 you are doing something wrong. The curl request is not pointing to mailcows Nginx then.

andryyy · 2019-04-06T20:30:30Z

Come on... your server or network can still be wrongly configured.

Copy-pasting the configs can also be fatal depending on the base install. You still need to know what you are doing. Other sites on your local Nginx/Apache can conflict with the RP site from the docs for example (there is more).

You can ask on IRC or buy support for hands-on.

chc-pr · 2019-04-09T10:18:38Z

I can now say that the fault is 100% with the MailCow installer - or at least its implementation of the letsencrypt module.

How do I know? Because I have just successfully set up an entirely new mailserver from scratch and installed the Letsencrypt certificate using the exact same hostname straight off first time try. Its working cert and all. I have done this starting with the exact same base install of Ubuntu 18.04.2 I used to try to load Mailcow.

It may be that the latest updates of Ubuntu have broken the installer, but it is now certainly broken. I suggest you fix your installer - or at least warn people of some other pre-requisites if it proves not to be broken but a NAT reflection requirement (which I asked about but you completely ignored). If its not that then its likely a typo symlink I suspect.

There were multiple clues in my posts, but you chose to assume I was an idiot and patronise me instead. If you go back and read MY (not the thread hijacker's) comments, I am sure you will get a clue as to where it is broken.

Happy hunting.

edit: Just for clarity - you also ignored my repeated confirmations that there was nothing else running on the VM onto which I was attempting to install MailCow - but again you chose to ignore that and assume I was an idiot. How the heck can there be a conflict with another Nginx/Apache if there IS NOT OTHER such installation.

Yes, I am pissed off with your attitude.

andryyy · 2019-04-09T16:01:51Z

Don't care.

Your forwarding is probably broken. It is not not your fault just because you can request a LE certificate without mailcow. oO You don't seem to understand the networking behind Docker. That's fine, you don't have to.

pbaeumel mentioned this issue Apr 5, 2019

Error in "mailcodockerized_mailcow-network" after ./update.sh - iptables #2500

Closed

JOduMonT mentioned this issue Apr 5, 2019

Yep; just another Lets Encrypt issue #2503

Closed

andryyy closed this as completed Apr 9, 2019

mailcow locked as off-topic and limited conversation to collaborators Apr 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to get Letsencrypt certificate #2498

Unable to get Letsencrypt certificate #2498

chc-pr commented Apr 4, 2019

andryyy commented Apr 5, 2019

pbaeumel commented Apr 5, 2019 •

edited

andryyy commented Apr 5, 2019

pbaeumel commented Apr 5, 2019 •

edited

chc-pr commented Apr 5, 2019 via email

andryyy commented Apr 5, 2019 via email

chc-pr commented Apr 6, 2019

andryyy commented Apr 6, 2019

chc-pr commented Apr 6, 2019 •

edited

andryyy commented Apr 6, 2019

chc-pr commented Apr 9, 2019 •

edited

andryyy commented Apr 9, 2019 •

edited

Unable to get Letsencrypt certificate #2498

Unable to get Letsencrypt certificate #2498

Comments

chc-pr commented Apr 4, 2019

andryyy commented Apr 5, 2019

pbaeumel commented Apr 5, 2019 • edited

andryyy commented Apr 5, 2019

pbaeumel commented Apr 5, 2019 • edited

chc-pr commented Apr 5, 2019 via email

andryyy commented Apr 5, 2019 via email

chc-pr commented Apr 6, 2019

404 Not Found

andryyy commented Apr 6, 2019

chc-pr commented Apr 6, 2019 • edited

andryyy commented Apr 6, 2019

chc-pr commented Apr 9, 2019 • edited

andryyy commented Apr 9, 2019 • edited

pbaeumel commented Apr 5, 2019 •

edited

pbaeumel commented Apr 5, 2019 •

edited

chc-pr commented Apr 6, 2019 •

edited

chc-pr commented Apr 9, 2019 •

edited

andryyy commented Apr 9, 2019 •

edited