I need help with failregex on Nginx Proxy Manager

[viriatusX]

Good evening,
I have Fail2Ban correctly installed and integrated with Nginx Proxy Manager (NPM).
My problem is that in NPM I have 20 proxy running. Not all of them have the same logs, and Fail2Ban some services does not catch the IP.
The Logs headers are common. Here is an example:

SERVICE 1

[02/May/2024:06:55:15 +0200] - 401 401 - GET https xxxxxxxx.com "/robots.txt" [Client xx8.xxx.242.44] [Length 61] [Gzip -] [Sent-to 192.168.10.165] "Mozilla/5.0 (compatible; Domains Project/1.3.7; +https://domainsproject.org)" "-"
[02/May/2024:06:55:17 +0200] - 401 401 - HEAD https xxxxxxxx.com "/" [Client 108.162.241.151] [Length 0] [Gzip -] [Sent-to 192.168.10.165] "Mozilla/5.0 (compatible; Domains Project/1.3.7; +https://domainsproject.org)" "-"
[02/May/2024:14:29:57 +0200] - 204 204 - OPTIONS https xxxxxxxx.com "/obsidian/" [Client xxxx.64.xx8.99] [Length 0] [Gzip -] [Sent-to 192.168.10.165] "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148" "-"
[02/May/2024:14:29:57 +0200] - 200 200 - GET https xxxxxxxx.com "/obsidian/" [Client xx2.64.xxxx.106] [Length 496] [Gzip -] [Sent-to 192.168.10.165] "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148" "-"

SERVICE 2

[02/May/2024:06:27:48 +0200] - 502 502 - GET https xxxxxxxx.com "/" [Client 1xx.71.1xxx.238] [Length 154] [Gzip -] [Sent-to 192.168.10.165] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "http://xxxxxxxx.com/"
[02/May/2024:06:53:05 +0200] - 502 502 - GET https xxxxxxxx.com "/" [Client 1x0.xx.25] [Length 154] [Gzip -] [Sent-to 192.168.10.165] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "http://xxxxxxxx.com"

SERVICE 3

[02/May/2024:11:47:52 +0200] - 200 200 - GET https xxxxxxxx.com "/" [Client 83.xx.2xx.53] [Length 14059] [Gzip 4.17] [Sent-to 192.168.10.165] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" "https://xxxxxxxx.com/"
[02/May/2024:11:47:52 +0200] - 204 204 - GET https xxxxxxxx.com "/members/api/member/" [Client 83.xx.xxx.53] [Length 0] [Gzip -] [Sent-to 192.168.10.165] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" "https://xxxxxxxx.com/"

My actual filter.d is:

[INCLUDES]

[Definition]

failregex = ^.* (405|404|403|401|\-) (405|404|403|401) - .* \[Client <HOST>\] \[Length .*\] .* \[Sent-to <F-CONTAINER>.*</F-CONTAINER>\] <F-USERAGENT>".*"</F-USERAGENT> .*$

ignoreregex = ^.* (404|\-) (404) - .*".*(\.png|\.txt|\.jpg|\.ico|\.js|\.css|\.ttf|\.woff|\.woff2)(/)*?" \[Client <HOST>\] \[Length .*\] ".*" .*$

With this filter, it intercepts some proxies, but not others.
What failregex could I apply to intercept everything?
As far as I can see in all the NPM logs, it repeats and is common:

GET - HOST - CLIENT - IP.

Can you help me?
Thank you.

[sebres]

With this filter, it intercepts some proxies, but not others.

It has nothing with proxies, but with response code - you trying to match 40x responses only, however "SERVICE 2" examples are 502 and "SERVICE 3" examples are 20x codes. Therefore they will never match (and should probably never did, at least 20x codes).

By the way, your RE is a bit vulnerable... Here is an example of anchored RE without catch-alls (matching 40x and 50x codes):

failregex = ^\s*(?:\[\] )?\S+ \S+ (?:[4-5]0\d) - \S+ \S+ \S+ "[^"]*" \[Client <ADDR>\] (?:\[\w+ [^\]]*\] )*\[Sent-to <F-CONTAINER>[^\]]*</F-CONTAINER>\] "<F-USERAGENT>[^"]*</F-USERAGENT>"

Also you wouldn't really need ignoreregex - you can rewrite the failregex using negative lookaheads or negative lookbehinds (if the intention is only to ignore certain extensions).
For instance this will ignore 404 for provided extensions:

- failregex = ^\s*(?:\[\] )?\S+ \S+ (?:[4-5]0\d) - \S+ \S+ \S+ "[^"]*" \[Client <ADDR>\] (?:\[\w+ [^\]]*\] )*\[Sent-to <F-CONTAINER>[^\]]*</F-CONTAINER>\] "<F-USERAGENT>[^"]*</F-USERAGENT>"
+ failregex = ^\s*(?:\[\] )?\S+ \S+ (?:(?P<nf>404)|[4-5]0(?!4)\d|504) - \S+ \S+ \S+ "(?(nf)/(?:[^/]*/)*[^\.]*(?!\.(?:png|txt|jpg|ico|js|css|ttf|woff|woff2)[^\."]*)|[^"]*)" \[Client <ADDR>\] (?:\[\w+ [^\]]*\] )*\[Sent-to <F-CONTAINER>[^\]]*</F-CONTAINER>\] "<F-USERAGENT>[^"]*</F-USERAGENT>"

But better would be to "fix" 404-references on all pages.
Also note https://github.com/fail2ban/fail2ban/wiki/Best-practice (especially section "Reduce parasitic log-traffic")

[viriatusX]

Thank you very much @sebres for helping me to improve the Failregex.
I understand that in the one you give me, it doesn't take into account the 20x and 30x requests either.

How do I add it and include them as well?

In short, I would like Fail2Ban to consider any request (even if it is 200 OK).

Thank you,

[sebres]

I would like Fail2Ban to consider any request (even if it is 200 OK).

Strange.
But here you go:

For the first RE simply replace:

- ... (?:[4-5]0\d) ...
+ ... \d+ ... 

For the second (with 404-conditional negative lookahead):

- ... (?:(?P<nf>404)|[4-5]0(?!4)\d|504) ...
+ ... (?:(?P<nf>404)|(?!404)\d+) ...

[viriatusX]

it's work! thanks :)