"Server seems not reachable" determined incorrectly?

[elieux]

I'm getting these errors:

mod_md.c(434): AH10037: server seems not reachable via http: and not reachable via https:
(20014)Internal error (specific information not available): md[example.com]: None of the ACME challenge methods configured for this domain are suitable. The http: challenge 'http-01' is disabled because the server seems not reachable on public port 80. The https: challenge 'tls-alpn-01' is disabled because the server seems not reachable on public port 443.The DNS challenge 'dns-01' is disabled because the directive 'MDChallengeDns01' is not configured.

(There's also a missing space before the last sentence, see https://github.com/icing/mod_md/blob/master/src/md_acme_drive.c#L591.)

The server isn't publicly accessible, but looking at the (hopefully) relevant detect_supported_protocols() there doesn't seem to be such a requirement, just a sanity check to see if the server is listening on the expected ports. I don't see why it should report these errors.

MDCAChallenges http-01 nor MDPortMap http:80 https:443 doesn't change the outcome.

My example config:

ServerRoot W:/temp/httpd

LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule md_module modules/mod_md.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule watchdog_module modules/mod_watchdog.so

DocumentRoot "W:/temp/httpd/htdocs"
<Directory "W:/temp/httpd/htdocs">
Require all granted
</Directory>

ServerAdmin admin@example.com
ErrorLog "logs/error.log"
LogLevel debug
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

MDCertificateAuthority https://acme.example.com/directory
MDomain example.com

Listen 192.168.88.100:80 192.168.88.100:443
<VirtualHost *:80 *:443>
	ServerName example.com
</VirtualHost>

Build params:

• Apache/2.4.43 (Win64) OpenSSL/1.1.1g
• Apache Lounge VS16 Server built: Apr 21 2020 16:23:13
• mod_ssl/2.4.43 compiled against Server: Apache/2.4.43, Library: OpenSSL/1.1.1g
• mod_md (v2.2.7)

[tlhackque]

The wording of the log message is a bit harsh, but I think the log message is correct. It is explained better in the PORTS PORTS PORTS section of the README.

It's a bit more than a sanity check, but it's not perfect. There really isn't a way to check for acessibility without invoking some external service - and there would be privacy complaints if mod_md did. Ultimately, it's up to the CA's validation agent. mod_md is trying to protect you - if it contacts the CA and the validation agent(s) can't reach the server, you can end up locked-out by that CA. (In the case of Let's Encrypt, for about a week.)

For the domain control verification to pass, the CA's agents (and the protocol) require that the client be reachable by its agents on specific ports. Port 80 for http-01, and Port 443 for tls-alpn-01. You can not use http-01 on port 443 (or with SSL), nor can you use tls-alpn-01 on port 80. There is no flexibility - that's the way it is. The reasoning is explained in the RFCs.

As the README explains, it IS possible that NAT or load balancers may map the external port to something else - e.g., the public facing port MUST be 80 for http-01, but httpd may actually be listening on 8080. That's why the message says "seems not reachable".

The server isn't publicly accessible

If the server isn't publicly accesible, mod_md won't obtain certificates for it using either http-01 or tls-alpn-01 (unless you're running your own ACME server). You can use dns-01 in this case - but as the log message explains, you need to confgure a MDChallengeDns01 program (usually a script) to install and remove the necessary TXT records in the public DNS.

<VirtualHost *:80 *:443>

This doesn't make sense. A VH is either plain HTTP, or TLS/SSL-enabled HTTPS. It can't be both. Either the VH is http - in which case you don't want to respond on 443, or SSL - in which case you need a SSLEngine on and don't want to respond (with this VH) on 80.

If this server is truly not publicly-accessible, you want to setup for dns-01.

If it is publicly accessible, you can setup either (or both) of the other responses - configuring a VH to properly match the ports.

MDCertificateAuthority https://acme.example.com/directory

Obfuscating your configuration to this extent makes it very difficult to be helpful. Are you really trying to run your own ACME CA? If not, the specific CA and directory URL would be helpful, especially when you get to the next step. I have recently seen some rather creative interpretations of the ACME RFCs...

If your server is publically accessible, there's really nothing to hide - once you have a certificate, anyone can tell where it came from. If it's blocked by a firewall, there's also nothing to hide :-)

[elieux]

I understand the first part. I didn't check HTTPS (which doesn't work with this config), but I did check HTTP and it works fine, so at least half of mod_md's assessment is still wrong.

As for the public accessibility, yes, I am testing my own ACME instance. Sorry for not mentioning this right away. I know how confusing it is to receive a request for advice with little to no context, yet I still make that mistake when I'm on the other side.

As for obfuscation, the domain name is not redacted because I'm afraid of my security, but because I'm using a company's domain name for my testing and I don't want to just throw it around publicly (and I assume they wouldn't want me to either).

<VirtualHost *:80 *:443>

This doesn't make sense. A VH is either plain HTTP, or TLS/SSL-enabled HTTPS. It can't be both. Either the VH is http - in which case you don't want to respond on 443, or SSL - in which case you need a SSLEngine on and don't want to respond (with this VH) on 80.

This is a syntax I've successfully been using for years. I can't find a good source, but I found some people using it as well, like here: https://serverfault.com/a/734263

Here is a working example config that serves HTTP and HTTPS for that virtual host correctly (modulo some variables):

ServerRoot W:/temp/httpd
[... same as before ...]
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

Listen 192.168.88.100:80
Listen 192.168.88.100:443
<VirtualHost *:443>
	SSLEngine On
	SSLCertificateFile "W:/temp/httpd/conf/cert.pem"
	SSLCertificateKeyFile "W:/temp/httpd/conf/key.pem"
</VirtualHost>
<VirtualHost *:80 *:443>
	ServerName example.com
</VirtualHost>

Aside from a missing virtual host enabling TLS for port 443 wholesale, the issue was my Listen directive. Apparently its syntax is Listen [IP-address:]portnumber [protocol], so my previous usage was incorrect (but still good enough for httpd to serve on HTTP).

I tried mod_md with the corrected Listen directives, but it didn't like this way of doing things:

mod_md.c(533): AH10168: example.com: no https server_rec found for example.com

After separating the virtual hosts, as you recommended, mod_md was happy. See this example config:

ServerRoot W:/temp/httpd
[... same as before ...]
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

MDCertificateAuthority https://acme.example.com/directory
MDomain example.com

Listen 192.168.88.100:80
Listen 192.168.88.100:443
<VirtualHost *:80>
	ServerName example.com
</VirtualHost>
<VirtualHost *:443>
	ServerName example.com
	SSLEngine On
</VirtualHost>

In summary:

• mod_md's sanity check is theoretically correct, but is stricter than httpd itself
• mod_md doesn't like combined HTTP+HTTPS virtual hosts, but httpd is okay with them

Do you think either of those deserve further attention?

And should I send a PR for the missing space? :)

[tlhackque]

I've never seen a(n intentiona) combined http/https VH before. I don't claim to be as expert as some, but to me that usage seems to be exploiting a bug (or loophole) in httpd. I'm not surprised that your serverfault.com reference reports that SERVER_PORT doesn't track the connection - there is a lot of code in httpd and mod_ssl that I've always read to assume that a VH is or is not "SSL".

The way I handle that is with an Include directive - I define the protocol independent configuration in a config file that is Included in both VHs. And the common SSL in another config file. I've also used Macros to handle minor differences...

Your working configuration isn't doing what you think. Virtual Host matching is described here. The short form is that an incoming request matches VHs on IP address/port. In this case, since both addresses are wildcards, first one it sees is what's used. So the :443 in your second VH is redundant. The reason things seem to work is (assuming you haven't elided more configuration) that both VHs are inheriting DocumentRoot from the default server. A request on 443 stops at the first VH, which has the SSL engine on. It never gets to the second VH. A requiest on 80 doesn't match the first VH, and stops at the second.

Arguably, httpd should complain that an IP address/port combination appears in more than one VH - or at least it would seem to be helpful if it did. But name-based VHs (described here and here) also need to be taken into account. Since you don't have a ServerName directive in your example, httpd defaults to "a FQDN derived from the system hostname". (Which is not recommended.) And since the IP addresses in your case are the same, the "first matching" rule applies. All of which is to say that detecting this ambiguity in httpd would be harder than it seems at first glance.

To the extent that httpd seems to be accepting an incorrect configuration - I'd argue that it does deserve further attention - but in the httpd bug tracker, not here. In the unlikely event that httpd disagrees with me and decides the split-personality configuration is supported, mod_md's behavior would be a valid issue here. (In my opinion.)

As for the mssing space - while I'm not the project owner, I am working on mod_md and have an on-going series of pull requests. I'll fix the missing space in my next edit. (Actually, I'm from school that says a period is followed by two spaces - hopefully not starting a religious war.)

Good luck with your own ACME isntance. If you're not using Boulder (developed and used by the Let's Encrypt folks), I'd be interested in knowing what you are using. As I mentioned, I've run into some very creative interpretations of the ACME RFCs recently, which have detourted my work. mod_md mostly tests against Boulder (and LE) - if other implementations are available, it would be good to include interoperability tests. If you're creating one in stealth mode, you can contact me using the yahoo.com domain.

[elieux]

The reason things seem to work is (assuming you haven't elided more configuration) that both VHs are inheriting DocumentRoot from the default server.

The configuration I posted was the one I tested with, almost verbatim. However, I have used this "trick" for more elaborate configurations, so for the sake of discussion, let's try this one:

[LoadModule ...]
[Directory ...]

ErrorLog "logs/error.log"
LogLevel debug
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

Listen 192.168.88.100:80
Listen 192.168.88.100:443

ServerRoot W:/temp/httpd
ServerName example.com
ServerAdmin admin@example.com
# DocumentRoot "W:/temp/httpd/htdocs"

<VirtualHost *:443>
	SSLEngine On
	SSLCertificateFile "W:/temp/httpd/conf/cert.pem"
	SSLCertificateKeyFile "W:/temp/httpd/conf/key.pem"
	# DocumentRoot "W:/temp/httpd/htdocs"
</VirtualHost>
<VirtualHost _default_:80 _default_:443>
	DocumentRoot "W:/temp/httpd/htdocs"
</VirtualHost>
<VirtualHost *:80 *:443>
	DocumentRoot "W:/temp/httpd/htdocs1"
	ServerName one.example.com
</VirtualHost>
<VirtualHost *:80 *:443>
	DocumentRoot "W:/temp/httpd/htdocs2"
	ServerName two.example.com
</VirtualHost>

The server responds to https?://one.example.com and https?://two.example.com using correct document roots. It responds correctly on http://192.168.88.100 as well, but not on https://192.168.88.100, unless I uncomment one of the extra DocumentRoot lines. This seems to confirm some of your suspicions about how httpd handles these combined virtual hosts, but not all of them. I'm not surprised I didn't notice this earlier, because I don't really use the default virtual host for anything.

[tlhackque]

It's more than "suspicion" - I provided links to the documentation on exactly how httpd resolves VHs, which is non-trivial. I think my summary of what happend in your case is accurate - for the general case, you need to (carefully) read that documentation. Note that when there's no ServerName, the result depends on what's in the DNS (and responses do extra DNS lookups). It requires several pages to describe and is quite involved - trying to do the right thing with forward and backward compatibility often is. IP virtual hosts, Named virtual hosts, and TLS/SNI have gone through several stages of evolution...

In any case, the behavior you've observed has been explained, and you have a simple way to get the desired results with mod_md. I'm happy with mod_md's behavior, so unless @icing has a different view, I think further discussion of what is "correct" for httpd is off-topic here.

Thanks for your report. I'm glad to have been able to help you move forward.

P.S. I have committed a fix for the missing space that should merge eventually, so you did effect some change :-)

[elieux]

Good luck with your own ACME isntance. If you're not using Boulder (developed and used by the Let's Encrypt folks), I'd be interested in knowing what you are using.

Oh, I forgot this part. Currently I'm using stock pebble with a possibility of small patches in the future.

[elieux]

I think further discussion of what is "correct" for httpd is off-topic here.

Makes sense.

Thanks for your report. I'm glad to have been able to help you move forward.

Thanks for the help.

[tlhackque]

Currently I'm using stock pebble with a possibility of small patches in the future.

Unless you are testing ACME clients (like mod_md, certbot, etc), Pebble isn't for you. Really.

Pebble is not a suitable production server. Among other things, it has no persistence and generates a new "CA" root certificate on every restart...so I don't see a future in that. The headline on the site reads:

Pebble is a small RFC 8555 ACME test server not suited for a production certificate authority

And the warnings - following a big flashing red JPG - continue after that. Please read them before investing time in that approach.

It's intended (and good) for testing clients like mod_md. It's useless for running your own CA (even an internal or hobbyist CA), and intentionally takes steps to keep it that way.

FYI: mod_md currently has trouble with Pebble- I've spent the past couple of weeks debugging "interesting" cases. It's done a good job of identifying cases where mod_md satisfies Boulder, but not the RFCs.

There are some fixes in this repo; more pending in mine. And I'm still chasing some behaviors, particularly when Pebble restarts without wiping out mod_md's persistent storage. My interest is in making mod_md more robust (and in testing certain other changes to mod_md that are WIP.)

Alternatives: It is possible to run a private Boulder instance (the mod_md test system has one packaged). Some CAs besides LE have adopted it. Others seem to be developing their own.