RF to allow arbitrary set of keys (not just HOST) to track the failures #67

yarikoptic opened this Issue Jul 29, 2012 · 15 comments


None yet
7 participants

yarikoptic commented Jul 29, 2012

From the mailing list:

> On 28-07-12 16:18, Gerry Magill wrote:
> > VoIP has to go through a SIP aware firewall in My scenario to resolve
> > NAT-traversal issues with the SIP protocol. So my SIP Registrar sees all
> > connections as coming from my SBCs IP. Each new distinct user appears to
> > me as traffic from a new port. This port binding never changes until my
> > SBC is rebooted.

> > I can identify SIP SPIT and SPAM using snort but blocking the whole IP
> > would kill my SBC.

> > So I am looking to black list IP port bindings I identify as being
> > malicious and release them after 3 days in jail.

So this use case would require to track not only by IP but by (IP, port) pair. We could implement this based on the named groups in the failregex (ATM only P<host> is utilized) and pass all those to the actions complementing the variables defined in Init and passed into the jail definition


yarikoptic commented May 21, 2013

moreover we should allow any failregex group to be used, possibly even without HOST. People request this feature from time to time and even I had a usecase for it (monitoring on server side transfer failures of rsyncd). In the simplest case only would have some semantics associated with it (IP address, ignoreip, etc)... alternatively we could also allow "ignoreGROUP" option to specify which hits to ignore for a GROUP


yarikoptic commented May 21, 2013

Example from the mailing list (from OCEANET - Cédric BASSAGET)

== jail.conf ==
enabled  = true
filter   = asterisk-internat
action   = asterisk-block-sippeer[name=asterisk-block-sippeer]
logpath  = /var/log/asterisk/call_agi.log
findtime = 60
maxretry = 10

== filter.d/asterisk-internat.conf ==
failregex = FAIL2BAN-out \| Call from <SIPPEER> to .*
ignoreregex =

== action.d/asterisk-block-sippeer.conf ==
actionstart =
actionstop =
actioncheck =
actionban = asterisk -rx "realtime update2 sippeers name <sippeer> deny"
actionunban = asterisk -rx "realtime update2 sippeers name <sippeer> deny NULL"

kwirk commented May 21, 2013

I agree that using regex groups is the best approach, e.g. as per mailing list post failregex = FAIL2BAN-out \| Call from (?P<sippeer>\S+) to .*.
Also need to check that group names are not ip, failures, time, matches, skiplines, br (I think that's all of them…).

I'm thinking that FailTickets should also then be changed to handle when ip/host is None, and other elements which expect this IP to be present, which is likely have quite a few potential knock on effects.

Also, now that I think about it, how do you work out that a set of FailTickets are of the same offender to tie them together to form a BanTicket. Previously they have been tied together with the fact the IP address is the same. Do all the elements of the regex groupdict need to be equal?

How should we handle if one or more of the groups return None? (Ideally you'd write the regex so this shouldn't happen…). Raise warning/error and do not generate FailTicket?

Lots of questions and issues, but I'm happy to have a go at implementing this 😄?


kwirk commented May 29, 2013

I've had another look at this and started playing with some ideas:

  • I dropped FailData, because it didn't seem that different to the data held within the FailTicket. I also changed the Tickets, such that a FailTicket represents a single fail from the filter, which can then be added to a FailuresTicket in the FailManager which represents one or more failures. This is then what is passed through the jail to Actions to create a BanTicket.
  • I also put in a scheme where regex group names which begin with a single underscore are ignored for matching FailTickets with each other, but could still be used in an action (last FailTicket overwrites _groups), and double underscore variables are dropped at the FailRegex point (e.g. __skiplines. The only exception is host which is ignored as IP fills this role.

Diff of changes at: https://gist.github.com/kwirk/5674184/24b4dce357659b7e3c82f5eaa8ffeed70ab1f531
Note finished yet, but wanted to get any thoughts and feedback to keep sure the approach is right. 😄


grooverdan commented May 30, 2013

An alternate to ignoreGroup is a failGROUP = definition in a jail that defines what the match is, Defaults to but really can be any group (or groups), as long as every failregex has all of them.

@kwirk I definitely need to hurry up with IPv6 because changes like this are going to be hard to merge. For IPv4 (#201) and IPv6 I almost need (very quickly):

Faildata[ (iptruncated,family,cidr) ] = [ (ip,family,match,time) , (ip,family,match,time), ... ]

So more generically:

Faildata[ (failGROUP1, failGROUP2,....) ] = [ (match,time, (failGROUP1, failGROUP2,....) ]

Even more generically would need a transformation function from match data to the unique key but we can leave this till later (I hope).


kwirk commented May 30, 2013

@grooverdan I think that the IPv6 is more critical, so to avoid any disruption I'd say it's best to sit on any changes related to this pull request for now, and wait for IPv6 and prefix changes to be completed

In respect to my diff on gist, I just had another thought that ip could also just become an element within the FailTicket group dictionary, and then equally the family and prefix could also just become part of the group dictionary? Changes on same gist, revision 2: https://gist.github.com/kwirk/5674184

grooverdan was assigned Aug 19, 2013


ghost commented Apr 28, 2014

Hi, is there any progress on this?


kwirk commented Apr 28, 2014

This was originally put on hold to avoid clashes with IPv6 work. However, I'm happy to resurrect my patches as I suspect the changes it was originally clashing with will need to be redefined quite a bit anyway as the code had moved on a fair amount. I would put this forward for 0.9.2 though, as we could do with releasing 0.9.1 as a primarily bug fix version (there are a fair few bug fixes since 0.9.0). Maybe even 1.0?

What do you think @yarikoptic @grooverdan?


yarikoptic commented Apr 28, 2014

On Mon, 28 Apr 2014, Steven Hiscocks wrote:

This was originally put on hold to avoid clashes with IPv6 work. However,
I'm happy to resurrect my patches as I suspect the changes it was
originally clashing with will need to be redefined quite a bit anyway as
the code had moved on a fair amount. I would put this forward for 0.9.2
though, as we could do with releasing 0.9.1 as a primarily bug fix version
(there are a fair few bug fixes since 0.9.0). Maybe even 1.0?

What do you think @yarikoptic @grooverdan?

agree that 0.9.1 must be primarily a bug fix. 1.0 or may be 0.10 might
be more appropriate for such a big RF/NF. Would be nice through indeed
if this feature could get the life.


@kwirk kwirk modified the milestone: 0.9.2, 0.9.1 May 7, 2014

@yarikoptic yarikoptic modified the milestone: 0.9.2, 0.9.3 Feb 14, 2015

leeclemens added the ipv6 label Jul 8, 2015

grooverdan removed their assignment Jul 9, 2015

@yarikoptic yarikoptic modified the milestone: 0.9.3, 0.9.4 Jul 11, 2015

@yarikoptic yarikoptic modified the milestone: 0.9.4 Jan 8, 2016

chetan-prime commented May 18, 2016 edited

I have a particular requirement with my email server to detect compromised users regardless of IP. I have a regex ready to detect when a user is sending emails and a script ready to disable that user temporarily.
All I need is a way to run the user disable script with fail2ban passing the email address captured using the regex.
Ideally this requires changes to fail2ban to allow grouping matches by a regex variable,or to allow setting HOST to a named regex match, so fail2ban can detect if the same user is sending emails too fast (irrespective of IP)

Will this be possible using the proposed changes to 0.9.4 ?

tguild commented May 27, 2016

Another person who needs more data from matched log line than just IPv4 address. Hope this gets released soon and surprised this has been open for nearly 4 years.


sebres commented May 30, 2016

@chetan-prime please test #1454

sebres closed this in #1454 May 31, 2016


sebres commented May 31, 2016

Issue can be reopened later:

  • if someone needs multiple groups (the "host" as ban-identifier should be accumulated from multiple "tags")
  • there is no unique ban identifier (user, pair ip:port, mac-address) available or there should not be unique: for example "banning" process does not really ban, but should just execute some action after X failures (for example just sends a mail to admin);

sebres removed the ipv6 label May 31, 2016

chetan-prime commented May 31, 2016 edited

Thank you @sebres , #1454 looks exactly like what I wanted. I'm testing and will report back in case of issues


sebres commented May 31, 2016

@chetan-prime, by merge in 0.10 I've currently realized, that I've forgotten to commit my last filter.py, that contains change, allows a new value 'raw' inside setUseDns.
Without this amend-fix usedns goes with fallback again to no, so the fail2ban will ignore such failures (because are not valid)

Please be sure by your tests, that you've actualized the master (and it contains b853474).

sebres referenced this issue Jun 19, 2016


Custom host definition expectations #1467

2 of 3 tasks complete

@the0ne the0ne pushed a commit to the0ne/fail2ban that referenced this issue Oct 12, 2016

root merged added option 'raw' to usedns from [gh-67] no-host ban #1454 d95cacc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment