Failregex for squid reverse proxy

[blackshadow1977]

Hello friends, today I have a problem with software fail2ban, I require oder block DDoS attacks to a website, which I generate many requests, I have managed to do in apache, but I can not do it with squid.

He explained how my architecture, I have two teams proxyreverso squid, behind them I have 5 apache web servers, these computers squid have them load balancing and caching our site. I require to block DDoS attacks, which occur at the site, it occurred to me thinking about counting the get, post, header but does not recognize me the date format fail2ban version 0.9.3 installed on a CentOS 6.8.

log fail2band

2016-07-25 11:04:19,411 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431779.860 40.77.167.21   4233 TCP_MISS/200 15587 GET http://subscriber.xxxxxxxxxx.com/Store/index.jsp? - DIRECT/192.168.1.6 text/html' but no valid date/time found for u' 1469431779.860 40.77.167.21   4233 TCP_MISS/200 15587 GET http://subscriber.xxxxxxxxxxx.com/Store/index.jsp? - DIRECT/192.168.1.6 text/html'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-07-25 11:04:19,417 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431779.916 144.76.105.35   5551 TCP_MISS/200 13351 GET http://www.educamericas.com/forward/13037 - DIRECT/192.168.5.80 text/html' but no valid date/time found for u' 1469431779.916 144.76.105.35   5551 TCP_MISS/200 13351 GET http://www.educamericas.com/forward/13037 - DIRECT/192.168.5.80 text/html'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-07-25 11:04:19,427 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431782.896 40.77.167.21   1976 TCP_MISS/200 46908 GET http://web4.xxxxxxxxxx.com/en/company-profile/infrastructure/ferrostaal-gmbh-ferrostaal/? - DIRECT/192.168.1.6 text/html' but no valid date/time found for u' 1469431782.896 40.77.167.21   1976 TCP_MISS/200 46908 GET http://web4.xxxxxxxxxxxx.com/en/company-profile/infrastructure/ferrostaal-gmbh-ferrostaal/? - DIRECT/192.168.1.6 text/html'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-07-25 11:04:19,445 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431785.565 157.55.39.27  10191 TCP_MISS/200 15270 GET http://www.xxxxxxxxxxx.com/cursos/Universidad-de-Valpara%C3%ADso - DIRECT/192.168.5.80 text/html' but no valid date/time found for u' 1469431785.565 157.55.39.27  10191 TCP_MISS/200 15270 GET http://www.xxxxxxxxxx.com/cursos/Universidad-de-Valpara%C3%ADso - DIRECT/192.168.5.80 text/html'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-07-25 11:04:19,466 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431785.617 157.55.39.27   6119 TCP_MISS/200 25724 GET http://admin.xxxxxxxxxxxx.com/articulos/actualidad/atencion-espana-y-latinoamerica-abierta-convocatoria-para-la-startup-competitio? - DIRECT/192.168.5.80 text/html' but no valid date/time found for u' 1469431785.617 157.55.39.27   6119 TCP_MISS/200 25724 GET http://admin.educamericas.com/articulos/actualidad/atencion-espana-y-latinoamerica-abierta-convocatoria-para-la-startup-competitio? - DIRECT/192.168.5.80 text/html'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-07-25 11:04:19,473 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431788.412 51.255.65.52   6812 TCP_MISS/200 15582 GET http://xxxxxxxxxxx.com/estudios/doctorados? - DIRECT/192.168.5.80 text/html' but no valid date/time found for u' 1469431788.412 51.255.65.52   6812 TCP_MISS/200 15582 GET http://educamericas.com/estudios/doctorados? - DIRECT/192.168.5.80 text/html'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-07-25 11:04:19,492 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431788.653 147.83.242.153      4 TCP_MISS/200 792 GET http://subscriber.xxxxxxxxxxxx.com/Subscriber/legacy/public_site - DIRECT/192.168.1.6 application/json' but no valid date/time found for u' 1469431788.653 147.83.242.153      4 TCP_MISS/200 792 GET http://subscriber.xxxxxxxxxxx.com/Subscriber/legacy/public_site - DIRECT/192.168.1.6 application/json'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.
2016-07-25 11:04:19,519 fail2ban.filter         [27761]: WARNING Found a match for u' 1469431788.657 147.83.242.153      7 TCP_MISS/200 642 POST http://subscriber.xxxxxxxxxx.com/Subscriber/site/public/ - DIRECT/192.168.1.6 application/json' but no valid date/time found for u' 1469431788.657 147.83.242.153      7 TCP_MISS/200 642 POST http://subscriber.xxxxxxxxxxxxxx.com/Subscriber/site/public/- DIRECT/192.168.1.6 application/json'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.

http-get-dos.conf

[Definition]

failregex = #^ <HOST> .*(GET|POST).*(http(s)?:\/\/).*
                <HOST> ((TCP)_[A-Z].*\d[0-9]{1,3}) (GET|POST) *.

ignoreregex =

Log Format Squid

logformat squid  %{%d-%b-%Y %H:%M:%S}tl  %>a %6tr %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt

out Log Squid

25-Jul-2016 11:32:45  206.48.240.166   2044 TCP_MISS/200 200727 GET http://www.xxxxxxx.com/en/news/mining/newmont-seeking-to-extend-yanacochas-life? - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:45  68.180.230.236      5 TCP_MISS/200 4023 GET http://www.xxxxxxx.com/TrialForm/img/logo.png;xxxxxxx_trial=715724143924da2b4cea1a010ff6 - DIRECT/192.168.1.6 image/png
 25-Jul-2016 11:32:45  190.108.95.6     53 TCP_MISS/200 341 GET http://member.xxxxxxx.com/cgi-bin/giemail? - DIRECT/192.168.2.5 image/gif
 25-Jul-2016 11:32:46  92.156.122.167   2014 TCP_MISS/200 22733 GET http://educamericas.com/category/keywords/start? - DIRECT/192.168.5.80 text/html
 25-Jul-2016 11:32:46  144.76.105.35   1404 TCP_MISS/200 13907 GET http://www.educamericas.com/forward/2097? - DIRECT/192.168.5.80 text/html
 25-Jul-2016 11:32:47  210.6.132.134   1222 TCP_MISS/200 47237 GET http://www.xxxxxxx.com/www/en/? - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:47  122.131.160.219      1 TCP_MISS/403 498 GET http://xxxxxxx.com/ - DIRECT/192.168.1.18 text/html
 25-Jul-2016 11:32:48  68.168.100.144   3970 TCP_MISS/200 195835 GET http://www.xxxxxxx.com/en/news/mining/underground-mining - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:49  207.46.13.45   4142 TCP_MISS/200 14309 GET http://www.xxxxxxx.com/Store/index.jsp? - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:49  210.6.132.134     47 TCP_MISS/200 545 GET http://www.xxxxxxx.com/www/notificaciones/muestra/ - DIRECT/192.168.1.6 application/json
 25-Jul-2016 11:32:49  40.77.167.21   2723 TCP_MISS/200 45258 GET http://www.xxxxxxx.com/es/reportajes/telecomunicaciones/bajo-la-lupa-at-t-la-gran-duda-en-brasil? - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:50  200.4.25.109     54 TCP_MISS/200 341 GET http://member.xxxxxxx.com/cgi-bin/giemail? - DIRECT/192.168.2.5 image/gif
 25-Jul-2016 11:32:50  68.168.100.72   2300 TCP_MISS/200 207632 GET http://www.xxxxxxx.com/en/news/mining/minera-escondida - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:51  207.46.13.45      1 TCP_MISS/404 562 GET http://xxxxxxx.com/company-profile/es/mapfre-la-centro-americana-sa-la-centro-americana - DIRECT/192.168.1.18 text/html
 25-Jul-2016 11:32:52  40.77.167.21   2021 TCP_MISS/200 44252 GET http://www.xxxxxxx.com/en/company-profile/oxbow-argentina-oxbow-argentina/ - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:53  68.168.100.144   4115 TCP_MISS/200 209070 GET http://www.xxxxxxx.com/en/news/mining/mining-industry - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:53  68.168.103.200   4968 TCP_MISS/200 211771 GET http://www.xxxxxxx.com/en/news/mining/coal-mining - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:56  112.111.160.75      0 TCP_MEM_HIT/301 465 HEAD http://events.xxxxxxx.com/ - NONE/- text/html
 25-Jul-2016 11:32:57  190.24.221.18      8 TCP_MISS/200 642 POST http://www.xxxxxxx.com/www/site/public/xxxxxxx - DIRECT/192.168.1.6 application/json
 25-Jul-2016 11:32:57  68.168.100.72   2095 TCP_MISS/200 193018 GET http://www.xxxxxxx.com/en/news/mining/mining-technology - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:58  68.168.100.144   4219 TCP_MISS/200 206879 GET http://www.xxxxxxx.com/en/news/mining/mining-supplies - DIRECT/192.168.1.6 text/html
 25-Jul-2016 11:32:58  210.6.132.134    315 TCP_MISS/200 4650 POST http://www.xxxxxxx.com/search/searchAutocomplete - DIRECT/192.168.1.6 text/plain

out jail

[root@srv-clus-1-public fail2ban]# fail2ban-client -d | grep 'http-get-dos'
['add', 'http-get-dos', 'auto']
['set', 'http-get-dos', 'usedns', 'warn']
['set', 'http-get-dos', 'addlogpath', '/var/log/squid/access.log', 'head']
['set', 'http-get-dos', 'maxretry', 150]
['set', 'http-get-dos', 'addignoreip', '127.0.0.1/8']
['set', 'http-get-dos', 'findtime', 300]
['set', 'http-get-dos', 'bantime', 1800]
['set', 'http-get-dos', 'addfailregex', '#^ <HOST> .*(GET|POST).*(http(s)?:\\/\\/).*']
['set', 'http-get-dos', 'addfailregex', '#^.*<HOST> .*(GET|POST).*']
['set', 'http-get-dos', 'addfailregex', '#\\d{0,10}.*\\d{0,3}\\ .*<HOST> ((TCP)_[A-Z].*\\d[0-9]{1,3}) (GET|POST) *.']
['set', 'http-get-dos', 'addfailregex', '<HOST> ((TCP)_[A-Z].*\\d[0-9]{1,3}) (GET|POST) *.']
['set', 'http-get-dos', 'addaction', 'iptables-multiport']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'actionban', '<iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'actionstop', '<iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>\n<iptables> -F f2b-<name>\n<iptables> -X f2b-<name>']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'actionstart', '<iptables> -N f2b-<name>\n<iptables> -A f2b-<name> -j <returntype>\n<iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'actionunban', '<iptables> -D f2b-<name> -s <ip> -j <blocktype>']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'actioncheck', "<iptables> -n -L <chain> | grep -q 'f2b-<name>[ \\t]'"]
['set', 'http-get-dos', 'action', 'iptables-multiport', 'iptables', 'iptables <lockingopt>']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/chain', 'INPUT']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/lockingopt', '']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'protocol', 'tcp']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'name', 'http-get-dos']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'chain', 'INPUT']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/__name__', 'Init']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/protocol', 'tcp']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/port', 'ssh']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/iptables', 'iptables <lockingopt>']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'lockingopt', '']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/name', 'default']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/blocktype', 'REJECT --reject-with icmp-port-unreachable']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'blocktype', 'REJECT --reject-with icmp-port-unreachable']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'returntype', 'RETURN']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'known/returntype', 'RETURN']
['set', 'http-get-dos', 'action', 'iptables-multiport', 'port', 'http,https']
['set', 'http-get-dos', 'addaction', 'sendmail-buffered']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'actionban', 'printf %b "`date`: <ip> (<failures> failures)\\n" >> <tmpfile>\nLINE=$( wc -l <tmpfile> | awk \'{ print $1 }\' )\nif [ $LINE -ge <lines> ]; then\nprintf %b "Subject: [Fail2Ban] <name>: summary from `uname -n`\nFrom: <sendername> <<sender>>\nTo: <dest>\\n\nHi,\\n\nThese hosts have been banned by Fail2Ban.\\n\n`cat <tmpfile>`\nRegards,\\n\nFail2Ban" | /usr/sbin/sendmail -f <sender> <dest>\nrm <tmpfile>\nfi']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'actionstop', 'if [ -f <tmpfile> ]; then\nprintf %b "Subject: [Fail2Ban] <name>: summary from `uname -n`\nFrom: <sendername> <<sender>>\nTo: <dest>\\n\nHi,\\n\nThese hosts have been banned by Fail2Ban.\\n\n`cat <tmpfile>`\nRegards,\\n\nFail2Ban" | /usr/sbin/sendmail -f <sender> <dest>\nrm <tmpfile>\nfi\nprintf %b "Subject: [Fail2Ban] <name>: stopped  on `uname -n`\nFrom: Fail2Ban <<sender>>\nTo: <dest>\\n\nHi,\\n\nThe jail <name> has been stopped.\\n\nRegards,\\n\nFail2Ban" | /usr/sbin/sendmail -f <sender> <dest>']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'actionstart', 'printf %b "Subject: [Fail2Ban] <name>: started on `uname -n`\nFrom: <sendername> <<sender>>\nTo: <dest>\\n\nHi,\\n\nThe jail <name> has been started successfully.\\n\nOutput will be buffered until <lines> lines are available.\\n\nRegards,\\n\nFail2Ban" | /usr/sbin/sendmail -f <sender> <dest>']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'actionunban', '']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'actioncheck', '']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'tmpfile', '/var/run/fail2ban/tmp-mail.txt']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'name', 'http-get-dos']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'known/sender', 'fail2ban']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'dest', 'support@bnamericas.com']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'known/__name__', 'Init']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'lines', '5']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'known/sendername', 'Fail2Ban']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'known/dest', 'root']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'sendername', 'Fail2Ban']
['set', 'http-get-dos', 'action', 'sendmail-buffered', 'sender', 'fail2ban']
['start', 'http-get-dos']

out log Fail2ban new

2016-07-25 11:40:53,874 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.0.45
2016-07-25 11:40:54,686 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 0 = ['0.0.0.0']
2016-07-25 11:40:54,687 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.0.0
2016-07-25 11:40:55,101 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 3 = ['0.0.0.3']
2016-07-25 11:40:55,102 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.0.3
2016-07-25 11:40:55,305 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 84 = ['0.0.0.84']
2016-07-25 11:40:55,306 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.0.84
2016-07-25 11:40:55,832 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 107 = ['0.0.0.107']
2016-07-25 11:40:55,832 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.0.107
2016-07-25 11:40:56,547 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 1 = ['0.0.0.1']
2016-07-25 11:40:56,550 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.0.1
2016-07-25 11:40:56,784 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 10618 = ['0.0.41.122']
2016-07-25 11:40:56,785 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.41.122
2016-07-25 11:40:56,819 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 5 = ['0.0.0.5']
2016-07-25 11:40:56,819 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.0.5
2016-07-25 11:40:57,304 fail2ban.filter         [27761]: WARNING Determined IP using DNS Lookup: 449 = ['0.0.1.193']
2016-07-25 11:40:57,304 fail2ban.filter         [27761]: INFO    [http-get-dos] Found 0.0.1.193

[root@srv-clus-1-public]# fail2ban-regex /var/log/squid/access.log /etc/fail2ban/filter.d/http-get-dos.conf 

Running tests
=============

Use   failregex filter file : http-get-dos, basedir: /etc/fail2ban
Use         log file : /var/log/squid/access.log
Use         encoding : ANSI_X3.4-1968


Results
=======

Failregex: 12406 total
|-  #) [# of hits] regular expression
|   4) [12406] <HOST> ((TCP)_[A-Z].*\d[0-9]{1,3}) (GET|POST) *.
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [12505] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
|  [111] Day(?P<_sep>[-/])Month(?P=_sep)(?:Year|Year2) 24hour:Minute:Second
`-

Lines: 45212 lines, 0 ignored, 12406 matched, 32806 missed [processed in 12.13 sec]
Missed line(s): too many to print.  Use --print-all-missed to print all 32806 lines

how I could resolve this error which gives me the date format?

thanks friends...

[sebres]

Your description is very chaotic, so confused.

Regarding the output of fail2ban-regex that has found 12406 matches (too few?)
This means that everything is fine with your date time pattern (or this log has different time patterns?)
Then please provide a log excerpt from access.log containing lines where fail2ban says "no valid date/time found for".
What did you mean with "out log fail2ban new"?

BTW. Your failregex is very, very bad (not anchored, greedy catch-alls, imprecise and dirty).
Better convenient expression for provided "out Log Squid" would be:

# this regex match any type of request:
#failregex = ^\s*<HOST>\s+\d+\s+TCP_[A-Z_]+/\d{1,3}\s+\d+\s+[A-Z]+

# this regex match only specified types of request (GET, POST, HEAD):
failregex  = ^\s*<HOST>\s+\d+\s+TCP_[A-Z_]+/\d{1,3}\s+\d+\s+(?:GET|POST|HEAD)

fail2ban-regex:

Use   failregex file : test-filters/squid.conf
Use         log file : test-filters/squid.txt
Use         encoding : UTF-8

Results
=======
Failregex: 22 total
|-  #) [# of hits] regular expression
|   1) [22] ^\s*<HOST>\s+\d+\s+TCP_[A-Z_]+/\d{1,3}\s+\d+\s+(?:GET|POST|HEAD)
`-
Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [22] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-
Lines: 22 lines, 0 ignored, 22 matched, 0 missed
[processed in 0.00 sec]

[blackshadow1977]

thank

[Francewhoa]

For others who need a F2B jail/failregex/filter for Squid, starting on 2014/Jan/22 F2B version 0.8.12 or more recent has a built-in filter for Squid. It is located at config/filter.d/squid.conf using [squid]

Thanks to Daniel Black & Roman Gelfand for their contributions