Permalink
Browse files

added support for pygeoip, added google map generation

  • Loading branch information...
1 parent 0b7e34c commit f2d29c72016aa3d204d193b31672b273c9976d40 @jvehent committed Jan 15, 2012
Showing with 270 additions and 88 deletions.
  1. +97 −50 README.md
  2. +173 −38 postscreen_stats.py
View
147 README.md
@@ -6,41 +6,75 @@ Run it against your postfix syslogs
Published under GPL v2
+Usage
+-------
+ postscreen_stats.py
+ parses postfix logs to compute statistics on postscreen activity
-Examples:
+ usage: postscreen_stats.py <-y|--year> <-r|--report|-f|--full>
+
+ <-a|--action> action filter with operators | and &
+ ex. 'PREGREET&DNSBL|HANGUP' => ((PREGREET and DNSBL) or HANGUP)
+ ex. 'HANGUP&DNSBL|PREGREET&DNSBL'
+ => ((HANGUP and DNSBL) or (PREGREET and DNSBL)
-Short report on full log file
----------------------------------
+ <-f|--file> log file to parse (default to /var/log/maillog)
+
+ <-g|--geoloc> /!\ slow ! ip geoloc against hostip.info (default disabled)
+
+ <--geofile> path to a maxmind geolitecity.dat. if specified, with the -g switch
+ the script uses the maxmind data instead of hostip.info (faster)
+
+ <-G> when using --geofile, use the pygeoip module instead of the GeoIP module
+
+ <-i|--ip> filters the results on a specific IP
+
+ <--mapdest> path to a destination HTML file that will display a Google Map of the result
+ /!\ Require the geolocation, preferably with --geofile
+
+ <-r|--report> report mode {short|full|ip} (default to short)
+
+ <-y|--year> select the year of the logs (default to current year)
+
+ Julien Vehent (http://1nw.eu/!j) - https://github.com/jvehent/Postscreen-Stats
+
+
+Basic usage
+--------------
+
+Generate a report form a syslog postfix log file.
+If you are parsing logs from a year that is not the current year, use the -y option to specify the year of the logs.
$ python postscreen_stats.py -f maillog.1 -r short -y 2011
- === Postscreen statistics ===
- 0 BLACKLISTED
- 33 COMMAND COUNT LIMIT
- 0 COMMAND LENGTH LIMIT
- 16 COMMAND PIPELINING
- 6 COMMAND TIME LIMIT
- 11010 CONNECT
- 536 DNSBL
- 503 HANGUP
- 42 NOQUEUE MAXCONN
- 0 NOQUEUE PORT BUSY
- 2258 NOQUEUE REJECT 450 (graylist)
- 1600 PASS NEW
- 8391 PASS OLD
- 239 PREGREET
- 84 WHITELISTED
- === Clients statistics ===
+ === unique clients/total postscreen actions ===
+ 2131/11010 CONNECT
+ 1/1 BARE NEWLINE
+ 30/33 COMMAND COUNT LIMIT
+ 13/16 COMMAND PIPELINING
+ 6/6 COMMAND TIME LIMIT
+ 463/536 DNSBL
+ 305/503 HANGUP
+ 12/15 NON-SMTP COMMAND
+ 1884/2258 NOQUEUE 450 deep protocol test reconnection
+ 1/42 NOQUEUE too many connections
+ 1577/1600 PASS NEW
+ 866/8391 PASS OLD
+ 181/239 PREGREET
+ 5/84 WHITELISTED
+
+ === clients statistics ===
4 avg. dnsbl rank
- 840 came back count
+ 505 blocked clients
2131 clients
- 32245 seconds avg. reco. delay
+ 840 reconnections
+ 32245.4285714 seconds avg. reco. delay
+
=== First reconnection delay (graylist) ===
delay| <10s |>10to30s| >30to1m| >1to5m | >5to30m|>30mto2h| >2hto5h|>5hto12h|>12to24h| >24h |
count|12 |21 |21 |196 |261 |88 |40 |29 |53 |119 |
% |1.4 |2.5 |2.5 |23 |31 |10 |4.8 |3.5 |6.3 |14 |
-
-
+
Get the statistics for a specific IP only
--------------------------------------------
@@ -59,34 +93,39 @@ Get the statistics for a specific IP only
Geo Localisation of blocked IPs
-----------------------------------
+There are 3 GeoIP modes:
+ 1. Use hostip.info online geoip service. This is free but slow and not very accurate
+ 2. Use Maxmind's GeoIP database. You can use either the free version of the DB from their website, or get a paid version.
-Use the '-g' switch to activate geolocalisation against hostip.info. At the moment, there are two big limitations to geolocalisation:
-1. It's slow ! Don't expect to get more than 2/3 IPs per second. So if you have 2000 IPs to geolocalise, it will take a while to run
-2. It only gives the Country of the IP. I didn't find the need to query the whole GPS data, but that's easy enough to change.
-
- $ python postscreen_stats.py -f maillog.1 -r short -g
-
- === Postscreen statistics ===
- 1 BARE NEWLINE
- 33 COMMAND COUNT LIMIT
- 16 COMMAND PIPELINING
- 6 COMMAND TIME LIMIT
- 11010 CONNECT
- 536 DNSBL
- 503 HANGUP
- 2258 NOQUEUE 450 deep protocol test reconnection
- 42 NOQUEUE too many connections
- 1600 PASS NEW
- 8391 PASS OLD
- 239 PREGREET
- 84 WHITELISTED
+To use hostip.info, just set the -g option.
+To use maxmind, set the --geofile to point to your Maxmind DB (ie. --geofile=/path/to/GeoIPCity.dat)
+By default, geofile use the GeoIP python module, but if you prefer to use pygeoip instead, set the -G option as well.
- === Clients statistics ===
- 4 avg. dnsbl rank
- 2131 clients
-
- === Blocked IPs per country ===
- [('XX', 238), ('US', 162), ('IN', 21), ('ID', 10), ('RU', 9), ('EU', 5), ('VN', 4), ('BR', 3), ('DE', 3), ('CO', 3), ('CA', 3), ('KR', 3), ('UK', 3), ('JP', 2), ('RO', 2), ('CN', 2), ('IT', 2), ('AR', 2), ('AU', 2), ('KZ', 2), ('MX', 2), ('FR', 1), ('BG', 1), ('BO', 1), ('NL', 1), ('PT', 1), ('TW', 1), ('TR', 1), ('TN', 1), ('LT', 1), ('PA', 1), ('PK', 1), ('PH', 1), ('PL', 1), ('CM', 1), ('IQ', 1), ('CZ', 1), ('ES', 1), ('SZ', 1), ('KE', 1), ('MW', 1), ('SA', 1), ('UA', 1)]
+ $ ./postscreen_stats.py -r short --geofile=../geoip/GeoIPCity.dat -G -f maillog.3 -y 2011
+
+ [....]
+
+ === Top 20 Countries of Blocked Clients ===
+ 167 (33.00%) United States
+ 59 (12.00%) India
+ 33 ( 6.50%) Russian Federation
+ 26 ( 5.10%) Indonesia
+ 23 ( 4.60%) Pakistan
+ 21 ( 4.20%) Vietnam
+ 20 ( 4.00%) China
+ 13 ( 2.60%) Brazil
+ 11 ( 2.20%) Korea, Republic of
+ 9 ( 1.80%) Belarus
+ 8 ( 1.60%) Turkey
+ 7 ( 1.40%) Iran, Islamic Republic of
+ 7 ( 1.40%) Ukraine
+ 6 ( 1.20%) Kazakstan
+ 6 ( 1.20%) Chile
+ 5 ( 0.99%) Italy
+ 5 ( 0.99%) Romania
+ 4 ( 0.79%) Poland
+ 4 ( 0.79%) Spain
+ 3 ( 0.59%) Afghanistan
Geo IP database installation
-----------------------------------
@@ -96,3 +135,11 @@ Using the MaxMind free database at http://www.maxmind.com/app/geolitecity
# aptitude install python-geoip
3. launch postscreen_stats with --geofile="/path/to/geolistcity.dat"
+Google Map of the blocked IPs
+-----------------------------------
+You can use the --geomap option to create an HTML file with a map of the blocked IPs.
+ $ ./postscreen_stats.py -f maillog.3 -r none -y 2011 --geofile=../geoip/GeoIPCity.dat -G --mapdest=postscreen_report_2012-01-15.html
+
+ Google map will be generated at postscreen_report_2012-01-15.html
+ using MaxMind GeoIP database from ../geoip/GeoIPCity.dat
+ Creating HTML map at postscreen_report_2012-01-15.html
Oops, something went wrong.

0 comments on commit f2d29c7

Please sign in to comment.