# "logencoding" specifies the encoding of the log files handled by the jail
# This is used to decode the lines from the log file.
# Typical examples: "ascii", "utf-8"
# auto: will use the system locale setting
logencoding = auto
logencoding = ASCII and logencoding= UTF-8 have this problem
Exim filter (exim.conf):
# Fail2Ban filter for exim
# This includes the rejection messages of exim. For spam and filter
# related bans use the exim-spam.conf
# Read common prefixes. If any customizations available -- read them from
before = exim-common.conf
failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$
^%(pid)s \w+ authenticator failed for (\S+ )?\(\S+\) \[<HOST>\]: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$
^%(pid)s %(host_info)sF=(<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (relay not permitted|Sender verify failed|Unknown user)\s*$
^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (connection from|"\S+") %(host_info)s(next )?input=".*"\s*$
^%(pid)s SMTP call from \S+ \[<HOST>\](:\d+)? (I=\[\S+\]:\d+ )?dropped: too many nonmail commands \(last was "\S+"\)\s*$
# DEV Notes:
# The %(host_info) defination contains a <HOST> match
# SMTP protocol synchronization error \([^)]*\) <- This needs to be non-greedy
# to void capture beyond ")" to avoid a DoS Injection vulnerabilty as input= is
# user injectable data.
# Author: Cyril Jaquier
# Daniel Black (rewrote with strong regexs)
mail subject: T="the price is € 100.00"
mail subject exim: T="the price is â~B¬100.00"
Problem Fail2Ban (/var/log/fail2ban.log):
2014-05-06 17:15:27,998 fail2ban.server.filter: WARNING Error decoding line from '/var/log/exim/mainlog' with 'US-ASCII': '2014-05-06 16:26:56 1WhgKd-000OMD-Gn <= H=  P=esmtps X=id= T="\xd0\xa1\xd0\xb5\xd0\xbc\xd0\xb8\xd0\xbd\xd0\xb0\xd1\x80 \xd0\xbf\xd0\xbe \xd0\xbf\xd1\x80\xd0\xb0\xd0\xba\xd1\x82\xd0\xb8\xd0\xba\xd0\xb5 \xd0\xba\xd0\xb0\xd0\xb4\xd1\x80\xd0\xbe\xd0\xb2\xd0\xbe\xd0\xb3\xd0\xbe \\n =?utf-8?Q?=D0=B4=D0=B5=D0=BB=D0=BE=D0=BF=D" from <> for
# Fail2Ban main configuration file
# Comments: use '#' for comment lines and ';' (following a space) for inline comments
# Changes: in most of the cases you should not modify this
# file, but provide customizations in fail2ban.local file, e.g.:
# loglevel = DEBUG
# Option: loglevel
# Notes.: Set the log level output.
# Values: [ LEVEL ] Default: ERROR
loglevel = NOTICE
# Option: logtarget
# Notes.: Set the log target. This could be a file, SYSLOG, STDERR or STDOUT.
# Only one log target can be specified.
# If you change logtarget from the default value and you are
# using logrotate -- also adjust or disable rotation in the
# corresponding configuration file
# (e.g. /etc/logrotate.d/fail2ban on Debian systems)
# Values: [ STDOUT | STDERR | SYSLOG | FILE ] Default: STDERR
logtarget = /var/log/fail2ban.log
How can I fix this problem?
I've tried to tidy up the issue text, which I've hopefully broken into the correct blocks
Fail2Ban 0.9.x supports Python 3, and hence the handling of encoding is required due to Python 3 differentiation between bytes and strings. This is why you don't see the issue 0.8.12.
The line is still being processed fully in Python 2, and in Python 3 is merely drops the characters it can't decode. Maybe the warning message should be clearer that this is the case…
The example error from fail2ban.log appears to be valid UTF-8:
In : b"\xd0\xa1\xd0\xb5\xd0\xbc\xd0\xb8\xd0\xbd\xd0\xb0\xd1\x80 \xd0\xbf\xd0\xbe \xd0\xbf\xd1\x80\xd0\xb0\xd0\xba\xd1\x82\xd0\xb8\xd0\xba\xd0\xb5 \xd0\xba\xd0\xb0\xd0\xb4\xd1\x80\xd0\xbe\xd0\xb2\xd0\xbe\xd0\xb3\xd0\xbe".decode('utf-8')
Out: 'Семинар по практике кадрового'
@dave670: Just to add, you can set the logencoding per jail if that helps.
If there is an issue with these warnings spamming the Fail2Ban log file, maybe there needs to be an option to suppress these warnings?
I'll close for now and hopefully the clearer message in #723 will help.