Augeas erroring out when encountering brackets in Include/IncludeOptional directive arguments

[AZed]

This was first seen with the certbot 0.10.2 package n Debian 9 ("Stretch"), but I got the same results with 0.11.1 from testing, 0.14.2 from experimental, and 0.16 from github. This might be a support candidate for #4799, but the fault doesn't actually appear to be in Augeas itself this time.

Certbot was invoked as:

certbot certonly --apache --cert-name blackbody.resonant.org -d blackbody.resonant.org -d www.resonant.org -d [... lots more ...]

It dies with the output:

An unexpected error occurred:
RuntimeError: ('Error during match procedure!', u"/files/etc/apache2/[c]onf.modules.d//*[label()=~regexp('[Vv][Ii][Rr][Tt][Uu][Aa][Ll][Hh][Oo][Ss][Tt]')]")

Full traceback is:

Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in
load_entry_point('certbot==0.14.2', 'console_scripts', 'certbot')()
File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 742, in main
return config.func(config, plugins)
File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 662, in certonly
installer, auth = plug_sel.choose_configurator_plugins(config, plugins, "certonly")
File "/usr/lib/python2.7/dist-packages/certbot/plugins/selection.py", line 187, in choose_configurator_plugins
installer = pick_installer(config, req_inst, plugins)
File "/usr/lib/python2.7/dist-packages/certbot/plugins/selection.py", line 32, in pick_installer
config, default, plugins, question, (interfaces.IInstaller,))
File "/usr/lib/python2.7/dist-packages/certbot/plugins/selection.py", line 77, in pick_plugin
verified.prepare()
File "/usr/lib/python2.7/dist-packages/certbot/plugins/disco.py", line 238, in prepare
return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)]
File "/usr/lib/python2.7/dist-packages/certbot/plugins/disco.py", line 120, in prepare
self._initialized.prepare()
File "/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py", line 196, in prepare
self.vhosts = self.get_virtual_hosts()
File "/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py", line 611, in get_virtual_hosts
(vhost_path, parser.case_i("VirtualHost"))))
File "/usr/lib/python2.7/dist-packages/augeas.py", line 415, in match
raise RuntimeError("Error during match procedure!", path)
RuntimeError: ('Error during match procedure!', u"/files/etc/apache2/[c]onf.modules.d//*[label()=~regexp('[Vv][Ii][Rr][Tt][Uu][Aa][Ll][Hh][Oo][Ss][Tt]')]")

This is, in fact, never going to match anything because Debian doesn't use /etc/apache2/conf.modules.d (that's a RedHat thing), and if I run augtool print, I can see a lot of entries being parsed from /files/etc/apache2/mods-available, but zero entries matching conf.modules.d. (Also, why is it looking for virtualhost lines there? Wouldn't those be in sites-available?)

I could in theory get most of those domains via --webroot instead of --apache, but that's 1) quite tedious with that many domains, and 2) doesn't actually work well for all of them, since some of the domains require authentication starting at the documentroot.

I am greatly looking forward to your wildcard certs, as I also control my own DNS, but since that is 6 months off still, I would greatly appreciate some ideas about how to get this fixed.

Additional weird thing that may or may not be a clue: a Debian Jessie box that I recently upgraded to Stretch was able to run certbot renew without any problems, also using the Apache plugin. I have absolutely no idea why it breaks on one but not the other. Both are configured from very similar Puppet rules, and the only major difference is that one has a lot more sites on it, and the one that was able to renew also lacks any augeas entry for conf.modules.d.

[SwartzCr]

Oh super weird!
I think what's happening is that the apache plugin is identifying your system as a redhat system. This is defined here: https://github.com/certbot/certbot/blob/master/certbot-apache/certbot_apache/constants.py
Which gets it's OS name from: https://github.com/certbot/certbot/blob/master/certbot-apache/certbot_apache/constants.py
So, can you tell me what you get when you cat /etc/os-release or try following the steps for the fallback OS classifier: https://github.com/certbot/certbot/blob/master/certbot/util.py#L406
If both of those look normal, it's possible that @joohoi has a sense of what's going on, since he wrote constants.py
Let me know if that helps!

[AZed]

/etc/os-release contains:

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Here's the output of the relevant python commands, gathered via IDLE:

>>> import os;
>>> import platform;
>>> platform.system()
'Linux'
>>> platform.release()
'4.9.0-3-amd64'
>>> platform.version()
'#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)'
>>> platform.system_alias(
        platform.system(),
        platform.release(),
        platform.version()
)
('Linux', '4.9.0-3-amd64', '#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)')

I note that os_ver = info[1] (line 428) is going to return the value of platform.release() in this, not platform.version(), so you're going to end up matching against 4.9.0-3-amd64, not the string that actually contains the word Debian. Is that possibly relevant?

[SwartzCr]

Quite possibly, @joohoi wrote that code so I'm curious to hear what he thinks about this

[joohoi]

I have a hunch that this is related to a weird Include or IncludeOptional statement somewhere in the configuration. If any way possible, could you post (possibly anonymized) version of your full apache configuration?

Even though it might be a Apache configuration issue, this is something that we should handle in the Certbot code, as it's obiviously a valid config for Apache itself.

[AZed]

I have a hunch that this is related to a weird Include or IncludeOptional statement somewhere in the configuration.

Bingo. Because this is a standardized config generated by Puppet valid for multiple operating systems, apache2.conf contains:

IncludeOptional [c]onf.modules.d/*.conf
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf

What's happening here is that this is a valid (if overly fancy) trick that works in Apache itself because [c]onf.modules.d forces a regular expression parse, which prevents it from failing in an IncludeOptional when the directory doesn't exist... but this is breaking an assumption either in Augeas or in the parser.

[joohoi]

Thanks for digging in @AZed ! We'll get it fixed at the Certbot end.

[SwartzCr]

Awesome - @joohoi do you want to leave this ticket open to track this, or close this and open a new ticket for that specific issue?

[joohoi]

This issue is still relevant, although not too many people have stumbled upon it apparently.