unable to start fail2ban-server when using custom sendmail-auth.local or sendmail-reject.local filters

[yolabingo]

Environment:

• First encountered in FreeBSD 12.1 py38-fail2ban-0.11.1_1
• reproduced in Linux on master branch
• Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

The presence of either a filter.d/sendmail-auth.local or filter.d/sendmail-reject.local file which contains a [Definition] section causes fail2ban to abort while loading its config during startup.

Steps to reproduce

• Enable either sendmail-auth or sendmail-reject - I'll use sendmail-auth for the example

printf "[sendmail-auth]\nenabled = true\n" > jail.local

• Create a custom sendmail-auth.local filter file

printf "[Definition]\nfailregex = foo\n" > filter.d/sendmail-auth.local

or even just

printf "[Definition]\n" > filter.d/sendmail-auth.local

• try to start fail2ban server

Expected behavior

fail2ban starts with customized sendmail-auth filter variable(s)

Observed behavior

fail2ban fails to start, producing this error:

Failed during configuration: Recursion limit exceeded in value substitution: option 'failregex' in section 'Definition' contains an interpolation key which cannot be substituted in 10 steps. Raw value: '^%(__prefix_line)s(\\S+ )?\\[(?:IPv6:<IP6>|<IP4>)\\]( \\(may be forged\\))?: possible SMTP attack: command=AUTH, count=\\d+$'

Any additional information

Due to this recent change to match longer Sendmail ID strings in the sendmail filter files:
__prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
I think python configparser is caught in endless recursion in
configparserinc.py: interpolate_some()

This can be resolved by creating a new variable instead of reusing __prefix_line in filter.d/sendmail-auth.conf:

sendmail_prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
# "w{14,20}" will give support for IDs from 14 up to 20 characters long
failregex = ^%(sendmail_prefix_line)s(\S+ )?\[(?:IPv6:<IP6>|<IP4>)\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$

I don't know enough about how configparser should behave in this unusual case to know if this is the best solution.

While debugging I added print statements to client/configparserinc.py_interpolate_some()

    class BasicInterpolationWithName(BasicInterpolation):

        def _interpolate_some(self, parser, option, accum, rest, section, map,
                              *args, **kwargs):
            if section and not (__name__ in map):
                map = map.copy()          # just to be safe
                map['__name__'] = section
                # try to wrap section options like %(section/option)s:
                print(f"section: {section}")
                print(f"accum: {accum}") 
                print(f"option : {option}")
                print(f"rest : {rest}")
                print(f"map: {map}")
                parser._map_section_options(section, option, rest, map)
                return super(BasicInterpolationWithName, self)._interpolate_some(
                    parser, option, accum, rest, section, map, *args, **kwargs)

which repeated these results before exiting with the Recursion limit exceeded error

section: Definition
accum: []
option : __prefix_line
rest : %(known/__prefix_line)s(?:\w{14,20}: )?
map: ChainMap({'__name__': 'Definition', 'known/__prefix_line': '%(known/__prefix_line)s(?:\\w{14,20}: )?'}, {'_daemon': '(?:sendmail|sm-(?:mta|acceptingconnections))', '__prefix_line': '%(known/__prefix_line)s(?:\\w{14,20}: )?', 'failregex': 'foo', 'ignoreregex': '', 'journalmatch': '_SYSTEMD_UNIT=sendmail.service'}, {'logtype': 'file', '_daemon': '\\S*', '__pid_re': '(?:\\[\\d+\\])', '__daemon_re': '[\\[\\(]?<_daemon>(?:\\(\\S+\\))?[\\]\\)]?:?', '__daemon_extra_re': '\\[ID \\d+ \\S+\\]', '__daemon_combs_re': '(?:<__pid_re>?:\\s+<__daemon_re>|<__daemon_re><__pid_re>?:?)', '__kernel_prefix': 'kernel:\\s?\\[ *\\d+\\.\\d+\\]:?', '__hostname': '\\S+', '__md5hex': '(?:[\\da-f]{2}:){15}[\\da-f]{2}', '__bsd_syslog_verbose': '<[^.]+\\.[^.]+>', '__vserver': '@vserver_\\S+', '__date_ambit': '(?:\\[\\])', '__prefix_line': '<lt_<logtype>/__prefix_line>', '__pam_auth': 'pam_unix', 'datepattern': '<lt_<logtype>/datepattern>'})

section: Definition
accum: []
option : __prefix_line
rest : %(known/__prefix_line)s(?:\w{14,20}: )?
map: ChainMap({'__name__': 'Definition', 'known/__prefix_line': '%(known/__prefix_line)s(?:\\w{14,20}: )?'}, {'_daemon': '(?:sendmail|sm-(?:mta|acceptingconnections))', '__prefix_line': '%(known/__prefix_line)s(?:\\w{14,20}: )?', 'failregex': 'foo', 'ignoreregex': '', 'journalmatch': '_SYSTEMD_UNIT=sendmail.service'}, {'logtype': 'file', '_daemon': '\\S*', '__pid_re': '(?:\\[\\d+\\])', '__daemon_re': '[\\[\\(]?<_daemon>(?:\\(\\S+\\))?[\\]\\)]?:?', '__daemon_extra_re': '\\[ID \\d+ \\S+\\]', '__daemon_combs_re': '(?:<__pid_re>?:\\s+<__daemon_re>|<__daemon_re><__pid_re>?:?)', '__kernel_prefix': 'kernel:\\s?\\[ *\\d+\\.\\d+\\]:?', '__hostname': '\\S+', '__md5hex': '(?:[\\da-f]{2}:){15}[\\da-f]{2}', '__bsd_syslog_verbose': '<[^.]+\\.[^.]+>', '__vserver': '@vserver_\\S+', '__date_ambit': '(?:\\[\\])', '__prefix_line': '<lt_<logtype>/__prefix_line>', '__pam_auth': 'pam_unix', 'datepattern': '<lt_<logtype>/datepattern>'})

Any customizations done to /etc/fail2ban/ configuration

jail.local:

[sendmail-auth]
enabled = true

filter.d/sendmail-auth.local:

[Definition]
failregex = foo

Relevant parts of /var/log/fail2ban.log file:

 +   89 7F0DC6658700 fail2ban.configparserinc  INFO  | configparserinc-20: read                 |   Loading files: ['/etc/fail2ban/filter.d/common.conf', '/etc/fail2ban/filter.d/sendmail-auth.conf', '/etc/fail2ban/filter.d/sendmail-auth.local']
 +   89 7F0DC6658700 fail2ban.configparserinc  TRACE | configparserinc-7 : _getSharedSCPWI      |     Shared file: /etc/fail2ban/filter.d/common.conf
 +   89 7F0DC6658700 fail2ban.configparserinc  TRACE | configparserinc-7 : _getSharedSCPWI      |     Shared file: /etc/fail2ban/filter.d/sendmail-auth.conf
 +   89 7F0DC6658700 fail2ban.configparserinc  TRACE | configparserinc-7 : _getSharedSCPWI      |     Shared file: /etc/fail2ban/filter.d/sendmail-auth.local
 +   90 7F0DC6658700 fail2ban                  ERROR | fail2bancmdline-40: readConfig           | Failed during configuration: Recursion limit exceeded in value substitution: option 'failregex' in section 'Definition' contains an interpolation key which cannot be substituted in 10 steps. Raw value: '^%(__prefix_line)s(\\S+ )?\\[(?:IPv6:<IP6>|<IP4>)\\]( \\(may be forged\\))?: possible SMTP attack: command=AUTH, count=\\d+$'

[yolabingo]

PR #2752

[sebres]

Fixed in e569281
Although your PR would also work, but in sense of fix, it is rather a workaround (solves the issue on sendmail-auth filter, but the origin of the problem remains).
Fix e569281 is ultimate, because it provides a general solution for every kind of issues like that.