Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RF to allow arbitrary set of keys (not just HOST) to track the failures #67

Closed
yarikoptic opened this issue Jul 29, 2012 · 18 comments
Closed

Comments

@yarikoptic
Copy link
Member

@yarikoptic yarikoptic commented Jul 29, 2012

From the mailing list:
http://sourceforge.net/mailarchive/forum.php?thread_name=50143F62.6050904%40whyscream.net&forum_name=fail2ban-users

> On 28-07-12 16:18, Gerry Magill wrote:
> > VoIP has to go through a SIP aware firewall in My scenario to resolve
> > NAT-traversal issues with the SIP protocol. So my SIP Registrar sees all
> > connections as coming from my SBCs IP. Each new distinct user appears to
> > me as traffic from a new port. This port binding never changes until my
> > SBC is rebooted.

> > I can identify SIP SPIT and SPAM using snort but blocking the whole IP
> > would kill my SBC.

> > So I am looking to black list IP port bindings I identify as being
> > malicious and release them after 3 days in jail.

So this use case would require to track not only by IP but by (IP, port) pair. We could implement this based on the named groups in the failregex (ATM only P<host> is utilized) and pass all those to the actions complementing the variables defined in Init and passed into the jail definition

@yarikoptic
Copy link
Member Author

@yarikoptic yarikoptic commented May 21, 2013

moreover we should allow any failregex group to be used, possibly even without HOST. People request this feature from time to time and even I had a usecase for it (monitoring on server side transfer failures of rsyncd). In the simplest case only would have some semantics associated with it (IP address, ignoreip, etc)... alternatively we could also allow "ignoreGROUP" option to specify which hits to ignore for a GROUP

@yarikoptic
Copy link
Member Author

@yarikoptic yarikoptic commented May 21, 2013

Example from the mailing list (from OCEANET - Cédric BASSAGET)

== jail.conf ==
[asterisk-international]
enabled  = true
filter   = asterisk-internat
action   = asterisk-block-sippeer[name=asterisk-block-sippeer]
logpath  = /var/log/asterisk/call_agi.log
findtime = 60
maxretry = 10

== filter.d/asterisk-internat.conf ==
[INCLUDES]
[Definition]
failregex = FAIL2BAN-out \| Call from <SIPPEER> to .*
ignoreregex =

== action.d/asterisk-block-sippeer.conf ==
[Definition]
actionstart =
actionstop =
actioncheck =
actionban = asterisk -rx "realtime update2 sippeers name <sippeer> deny 0.0.0.0/0.0.0.0"
actionunban = asterisk -rx "realtime update2 sippeers name <sippeer> deny NULL"

@kwirk
Copy link
Contributor

@kwirk kwirk commented May 21, 2013

I agree that using regex groups is the best approach, e.g. as per mailing list post failregex = FAIL2BAN-out \| Call from (?P<sippeer>\S+) to .*.
Also need to check that group names are not ip, failures, time, matches, skiplines, br (I think that's all of them…).

I'm thinking that FailTickets should also then be changed to handle when ip/host is None, and other elements which expect this IP to be present, which is likely have quite a few potential knock on effects.

Also, now that I think about it, how do you work out that a set of FailTickets are of the same offender to tie them together to form a BanTicket. Previously they have been tied together with the fact the IP address is the same. Do all the elements of the regex groupdict need to be equal?

How should we handle if one or more of the groups return None? (Ideally you'd write the regex so this shouldn't happen…). Raise warning/error and do not generate FailTicket?

Lots of questions and issues, but I'm happy to have a go at implementing this 😄?

@kwirk
Copy link
Contributor

@kwirk kwirk commented May 29, 2013

I've had another look at this and started playing with some ideas:

  • I dropped FailData, because it didn't seem that different to the data held within the FailTicket. I also changed the Tickets, such that a FailTicket represents a single fail from the filter, which can then be added to a FailuresTicket in the FailManager which represents one or more failures. This is then what is passed through the jail to Actions to create a BanTicket.
  • I also put in a scheme where regex group names which begin with a single underscore are ignored for matching FailTickets with each other, but could still be used in an action (last FailTicket overwrites _groups), and double underscore variables are dropped at the FailRegex point (e.g. __skiplines. The only exception is host which is ignored as IP fills this role.

Diff of changes at: https://gist.github.com/kwirk/5674184/24b4dce357659b7e3c82f5eaa8ffeed70ab1f531
Note finished yet, but wanted to get any thoughts and feedback to keep sure the approach is right. 😄

@grooverdan
Copy link
Contributor

@grooverdan grooverdan commented May 30, 2013

An alternate to ignoreGroup is a failGROUP = definition in a jail that defines what the match is, Defaults to but really can be any group (or groups), as long as every failregex has all of them.

@kwirk I definitely need to hurry up with IPv6 because changes like this are going to be hard to merge. For IPv4 (#201) and IPv6 I almost need (very quickly):

Faildata[ (iptruncated,family,cidr) ] = [ (ip,family,match,time) , (ip,family,match,time), ... ]

So more generically:

Faildata[ (failGROUP1, failGROUP2,....) ] = [ (match,time, (failGROUP1, failGROUP2,....) ]

Even more generically would need a transformation function from match data to the unique key but we can leave this till later (I hope).

@kwirk
Copy link
Contributor

@kwirk kwirk commented May 30, 2013

@grooverdan I think that the IPv6 is more critical, so to avoid any disruption I'd say it's best to sit on any changes related to this pull request for now, and wait for IPv6 and prefix changes to be completed

In respect to my diff on gist, I just had another thought that ip could also just become an element within the FailTicket group dictionary, and then equally the family and prefix could also just become part of the group dictionary? Changes on same gist, revision 2: https://gist.github.com/kwirk/5674184

@ghost
Copy link

@ghost ghost commented Apr 28, 2014

Hi, is there any progress on this?

@kwirk
Copy link
Contributor

@kwirk kwirk commented Apr 28, 2014

This was originally put on hold to avoid clashes with IPv6 work. However, I'm happy to resurrect my patches as I suspect the changes it was originally clashing with will need to be redefined quite a bit anyway as the code had moved on a fair amount. I would put this forward for 0.9.2 though, as we could do with releasing 0.9.1 as a primarily bug fix version (there are a fair few bug fixes since 0.9.0). Maybe even 1.0?

What do you think @yarikoptic @grooverdan?

@yarikoptic
Copy link
Member Author

@yarikoptic yarikoptic commented Apr 28, 2014

On Mon, 28 Apr 2014, Steven Hiscocks wrote:

This was originally put on hold to avoid clashes with IPv6 work. However,
I'm happy to resurrect my patches as I suspect the changes it was
originally clashing with will need to be redefined quite a bit anyway as
the code had moved on a fair amount. I would put this forward for 0.9.2
though, as we could do with releasing 0.9.1 as a primarily bug fix version
(there are a fair few bug fixes since 0.9.0). Maybe even 1.0?

What do you think @yarikoptic @grooverdan?

agree that 0.9.1 must be primarily a bug fix. 1.0 or may be 0.10 might
be more appropriate for such a big RF/NF. Would be nice through indeed
if this feature could get the life.

Cheers!

@kwirk kwirk added this to the 0.9.2 milestone May 7, 2014
@kwirk kwirk removed this from the 0.9.1 milestone May 7, 2014
@yarikoptic yarikoptic removed this from the 0.9.2 milestone Feb 14, 2015
@yarikoptic yarikoptic added this to the 0.9.3 milestone Feb 14, 2015
@yarikoptic yarikoptic added this to the 0.9.3 milestone Feb 14, 2015
@yarikoptic yarikoptic removed this from the 0.9.2 milestone Feb 14, 2015
@leeclemens leeclemens added the ipv6 label Jul 8, 2015
@grooverdan grooverdan removed their assignment Jul 9, 2015
@yarikoptic yarikoptic removed this from the 0.9.3 milestone Jul 11, 2015
@yarikoptic yarikoptic added this to the 0.9.4 milestone Jul 11, 2015
@yarikoptic yarikoptic added this to the 0.9.4 milestone Jul 11, 2015
@yarikoptic yarikoptic removed this from the 0.9.3 milestone Jul 11, 2015
@yarikoptic yarikoptic removed this from the 0.9.4 milestone Jan 8, 2016
@yarikoptic yarikoptic removed this from the 0.9.4 milestone Jan 8, 2016
@chetan-prime
Copy link

@chetan-prime chetan-prime commented May 18, 2016

I have a particular requirement with my email server to detect compromised users regardless of IP. I have a regex ready to detect when a user is sending emails and a script ready to disable that user temporarily.
All I need is a way to run the user disable script with fail2ban passing the email address captured using the regex.
Ideally this requires changes to fail2ban to allow grouping matches by a regex variable,or to allow setting HOST to a named regex match, so fail2ban can detect if the same user is sending emails too fast (irrespective of IP)
#1110

Will this be possible using the proposed changes to 0.9.4 ?

@tguild
Copy link

@tguild tguild commented May 27, 2016

Another person who needs more data from matched log line than just IPv4 address. Hope this gets released soon and surprised this has been open for nearly 4 years.

@sebres
Copy link
Contributor

@sebres sebres commented May 30, 2016

@chetan-prime please test #1454

@sebres sebres closed this in #1454 May 31, 2016
sebres added a commit that referenced this issue May 31, 2016
[gh-67] no-host ban
@sebres
Copy link
Contributor

@sebres sebres commented May 31, 2016

Issue can be reopened later:

  • if someone needs multiple groups (the "host" as ban-identifier should be accumulated from multiple "tags")
  • there is no unique ban identifier (user, pair ip:port, mac-address) available or there should not be unique: for example "banning" process does not really ban, but should just execute some action after X failures (for example just sends a mail to admin);

@sebres sebres removed the ipv6 label May 31, 2016
@chetan-prime
Copy link

@chetan-prime chetan-prime commented May 31, 2016

Thank you @sebres , #1454 looks exactly like what I wanted. I'm testing and will report back in case of issues

@sebres
Copy link
Contributor

@sebres sebres commented May 31, 2016

@chetan-prime, by merge in 0.10 I've currently realized, that I've forgotten to commit my last filter.py, that contains change, allows a new value 'raw' inside setUseDns.
Without this amend-fix usedns goes with fallback again to no, so the fail2ban will ignore such failures (because are not valid)

Please be sure by your tests, that you've actualized the master (and it contains b853474).

@jerrykan
Copy link

@jerrykan jerrykan commented Jun 10, 2020

* if someone needs multiple groups (the "host" as ban-identifier should be accumulated from multiple "tags")

Sorry to comment in a closed issue, but was the ability to combine multiple groups/tags into a ban-identifier ever implemented? After searching everywhere this seems to be the only place that mentions it.

I have a use-case with the squid proxy where I want to match on the client IP and the requested domain. The log line looks something like:

1591753818.254      0 192.168.1.30 TCP_DENIED/407 3978 CONNECT www.example.com:443 - HIER_NONE/- text/html

Because of how squid authentication works it is expected that there will be a TCP_DENIED/407 (failure) response followed by a TCP_MISS/200 (success). However occasionally a client doesn't work well with the proxy and keeps making requests resulting in TCP_DENIED/407. I would like to match on both the client IP and the requested domain as a unique pair and ban the IP if there are persistent TCP_DENIED/407 for the same domain. Without combining multiple tags this doesn't seem possible.

@sebres
Copy link
Contributor

@sebres sebres commented Jun 10, 2020

Sorry to comment in a closed issue, but was the ability to combine multiple groups/tags into a ban-identifier ever implemented?

Conditionally. You cannot use multiple <HOST> tags (as well as combine <HOST> with <ADDR> or <SUBNET>), but you can use <ADDR> (for IPv4 or IPv6) together with <DNS> (for hostname) or even <F-ID/>, just ID has higher precedence as IPs that have higher precedence as DNS. So single tag with higher precedence is always preferred if it is found.
And only DNS (if no IP there) would expand it to several tickets for every IP retrieved by dns-lookup.

So basically it is quasi impossible to obtain multiple IDs from single line (excepting the above mentioned case with DNS).

Although we have introduced capturing alternate groups with alt_ prefix, but it will neither affect ID of the ticket, nor expand the ticket for every alternate value found.

I have a use-case with the squid proxy where I want to match on the client IP and the requested domain.

It depends what you want to do here. Although you will generate single ticket here (if use IP) or multiple tickets if use DNS, If you have your own action you can obtain every token you had specified in filter (regex).

I would like to match on both the client IP and the requested domain as a unique pair and ban the IP if there are persistent TCP_DENIED/407 for the same domain.

It looks like you want implement own multi-line handling identified failure.
So the IP is a ticket ID, and IP with DNS is the multi-line failure ID.
I guess it is hardly possible, but you should provide log excerpt (with several messages) illustrating a failure and normal case (that should be not detected as a failure).

As an example see our current sshd-filter (multiline failure ID <F-MLFID>...</F-MLFID> identifying the session and <HOST> identifying ticket that gets banned). But as already said, it depends. So provide the log-excerpt either.

But normally to avoid banning of whole proxy, so to ban proxying requests properly (again not the proxy but rather real-IP (X-Forwarded-For) or session or something identifying user), one have to implement custom action that would prohibit it (e. g. add to blacklist) inside of the proxy (not using net-filter like iptables).
For example see our action that can banning in nginx - action.d/nginx-block-map.conf.

@jerrykan
Copy link

@jerrykan jerrykan commented Jun 11, 2020

@sebres thanks for taking the time to respond to my query. I've created a new issue to continue the discussion so I'm not spamming a bunch of people subscribed to a closed issue.

see: #2755

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants