Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Geoip #28

Closed
wants to merge 2 commits into from

1 participant

@msimerson
Owner

added GeoIP City support, continent, distance

wrote lots of succinct POD, and lots more tests.

This plugin saves geographic information in the following connection
notes:

     geoip_country      - 2 char country code
     geoip_country_name - full english name of country
     geoip_continent    - 2 char continent code
     geoip_distance     - distance in kilometers

   And adds entries like this to your logs:

     (connect) ident::geoip: US, United States, NA,    1319 km
     (connect) ident::geoip: IN, India, AS,    13862 km
     (connect) ident::geoip: fail: no results
     (connect) ident::geoip: CA, Canada, NA,   2464 km
@msimerson msimerson closed this
@msimerson msimerson deleted the msimerson:geoip branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 6, 2012
  1. @msimerson

    geoip: added geodesic distance calculations

    msimerson authored
    which require the GeoIP city data and Math::Complex
  2. @msimerson
This page is out of date. Refresh to see the latest.
Showing with 387 additions and 17 deletions.
  1. +270 −17 plugins/ident/geoip
  2. +117 −0 t/plugin_tests/ident/geoip
View
287 plugins/ident/geoip
@@ -1,17 +1,102 @@
#!perl -w
+=head1 NAME
+
+geoip - provide geographic information about mail senders.
+
=head1 SYNOPSIS
-This plugin uses MaxMind's GeoIP service and the Geo::IP perl module to
-do a lookup on incoming connections and record the country of origin.
+Use MaxMind's GeoIP databases and the Geo::IP perl module to report geographic
+information about incoming connections.
+
+=head1 DESCRIPTION
+
+This plugin saves geographic information in the following connection notes:
+
+ geoip_country - 2 char country code
+ geoip_country_name - full english name of country
+ geoip_continent - 2 char continent code
+ geoip_distance - distance in kilometers
+
+And adds entries like this to your logs:
+
+ (connect) ident::geoip: US, United States, NA, 1319 km
+ (connect) ident::geoip: IN, India, AS, 13862 km
+ (connect) ident::geoip: fail: no results
+ (connect) ident::geoip: CA, Canada, NA, 2464 km
+ (connect) ident::geoip: US, United States, NA, 2318 km
+ (connect) ident::geoip: PK, Pakistan, AS, 12578 km
+ (connect) ident::geoip: TJ, Tajikistan, AS, 11965 km
+ (connect) ident::geoip: AT, Austria, EU, 8745 km
+ (connect) ident::geoip: IR, Iran, Islamic Republic of, AS, 12180 km
+ (connect) ident::geoip: BY, Belarus, EU, 9030 km
+ (connect) ident::geoip: CN, China, AS, 11254 km
+ (connect) ident::geoip: PA, Panama, NA, 3163 km
+
+Calculating the distance has three prerequsites:
+
+ 1. The MaxMind city database (free or subscription)
+ 2. The Math::Complex perl module
+ 3. The IP address of this mail server (see CONFIG)
+
+Other plugins can utilize the geographic notes to alter the
+connection, reject, greylist, etc.
+
+=head1 CONFIG
+
+The following options can be appended in this plugins config/plugins entry.
+
+=head2 distance <IP Address>
+
+Enables geodesic distance calculation. Will calculate the distance "as the
+crow flies" from the remote mail server. Accepts a single argument, the IP
+address to calculate the distance from. This will typically be the public
+IP of your mail server.
+
+ ident/geoip [ distance 192.0.1.5 ]
+
+Default: none. (no distance calculations)
+
+=head2 db_dir </path/to/GeoIP>
+
+The path to the GeoIP database directory.
+
+ ident/geoip [ db_dir /etc/GeoIP ]
+
+Default: /usr/local/share/GeoIP
+
+=head1 LIMITATIONS
+
+The distance calculations are more concerned with being fast than accurate.
+The MaxMind location data is collected from whois and is of limited accuracy.
+MaxMind offers more accurate data for a fee.
+
+For distance calculations, the earth is considered a perfect sphere. In
+reality, it is not. Accuracy should be within 1%.
+
+This plugin does not update the GeoIP databases. You may want to.
+
+=head1 CHANGES
+
+2012-06 - Matt Simerson - added GeoIP City support, continent, distance
-Thats all it does.
+2012-05 - Matt Simerson - added geoip_country_name note, added tests
-It logs the 2 char country code to connection note I<geoip_country>.
-It logs the country name to the connection note I<geoip_country_name>.
+=head1 SEE ALSO
-Other plugins can use that info to do things to the connection, like
-reject or greylist.
+MaxMind: http://www.maxmind.com/
+
+Databases: http://geolite.maxmind.com/download/geoip/database
+
+It may become worth adding support for Geo::IPfree, which uses another
+data source: http://software77.net/geo-ip/
+
+=head1 ACKNOWLEDGEMENTS
+
+Stevan Bajic, the DSPAM author, who suggested SNARE, which describes using
+geodesic distance to determine spam probability. The research paper on SNARE
+can be found here:
+http://smartech.gatech.edu/bitstream/handle/1853/25135/GT-CSE-08-02.pdf
=cut
@@ -19,10 +104,16 @@ use strict;
use warnings;
use Qpsmtpd::Constants;
-#use Geo::IP; # eval'ed in register()
+#use Geo::IP; # eval'ed in register()
+#use Math::Trig; # eval'ed in set_distance_gc
sub register {
- my $self = shift;
+ my ($self, $qp ) = shift, shift;
+
+ $self->log(LOGERROR, "Bad arguments") if @_ % 2;
+ $self->{_args} = { @_ };
+ $self->{_args}{db_dir} ||= '/usr/local/share/GeoIP';
+
eval 'use Geo::IP';
if ( $@ ) {
warn "could not load Geo::IP";
@@ -30,30 +121,192 @@ sub register {
return;
};
+# Note that opening the GeoIP DB only in register has caused problems before:
+# https://github.com/smtpd/qpsmtpd/commit/29ea9516806e9a8ca6519fcf987dbd684793ebdd#plugins/ident/geoip
+# Opening the DB anew for every connection is horribly inefficient.
+# Instead, attempt to reopen upon connect if the DB connection fails.
+ $self->open_geoip_db();
+
+ $self->init_my_country_code();
+
$self->register_hook( 'connect', 'connect_handler' );
};
sub connect_handler {
my $self = shift;
- my $geoip = Geo::IP->new();
- my $remote_ip = $self->qp->connection->remote_ip;
+ # reopen the DB if Geo::IP failed due to DB update
+ $self->open_geoip_db();
- my $c_code = $geoip->country_code_by_addr( $remote_ip ) or do {
+ my $c_code = $self->set_country_code() or do {
$self->log( LOGINFO, "fail: no results" );
return DECLINED;
};
+ $self->qp->connection->notes('geoip_country', $c_code);
- my $c_name = $geoip->country_name_by_addr( $remote_ip );
- if ( $c_name ) {
- $self->connection->notes('geoip_country_name', $c_name);
- };
+ my $c_name = $self->set_country_name();
+ my ($continent_code, $distance);
- $self->connection->notes('geoip_country', $c_code);
+ if ( $self->{_my_country_code} ) {
+ $continent_code = $self->set_continent( $c_code );
+ $distance = $self->set_distance_gc();
+ };
my $message = $c_code;
$message .= ", $c_name" if $c_name;
+ $message .= ", $continent_code" if $continent_code && $continent_code ne '--';
+ $message .= ", \t$distance km" if $distance;
$self->log(LOGINFO, $message);
return DECLINED;
}
+
+sub open_geoip_db {
+ my $self = shift;
+
+ # this might detect if the DB connection failed. If not, this is where
+ # to add more code to do it.
+ return if ( defined $self->{_geoip_city} || defined $self->{_geoip} );
+
+ # The methods for using GeoIP work differently for the City vs Country DB
+ # save the handles in different locations
+ my $db_dir = $self->{_args}{db_dir};
+ foreach my $db ( qw/ GeoIPCity GeoLiteCity / ) {
+ if ( -f "$db_dir/$db.dat" ) {
+ $self->log(LOGDEBUG, "using db $db");
+ $self->{_geoip_city} = Geo::IP->open( "$db_dir/$db.dat" );
+ }
+ };
+
+ # can't think of a good reason to load country if city data is present
+ if ( ! $self->{_geoip_city} ) {
+ $self->log(LOGDEBUG, "using default db");
+ $self->{_geoip} = Geo::IP->new(); # loads default Country DB
+ };
+};
+
+sub init_my_country_code {
+ my $self = shift;
+ my $ip = $self->{_args}{distance} or return;
+ $self->{_my_country_code} = $self->get_country_code( $ip );
+};
+
+sub set_country_code {
+ my $self = shift;
+ return $self->get_country_code_gc() if $self->{_geoip_city};
+ my $remote_ip = $self->qp->connection->remote_ip;
+ my $code = $self->get_country_code();
+ $self->qp->connection->notes('geoip_country', $code);
+ return $code;
+};
+
+sub get_country_code {
+ my $self = shift;
+ my $ip = shift || $self->qp->connection->remote_ip;
+ return $self->get_country_code_gc( $ip ) if $self->{_geoip_city};
+ return $self->{_geoip}->country_code_by_addr( $ip );
+};
+
+sub get_country_code_gc {
+ my $self = shift;
+ my $ip = shift || $self->qp->connection->remote_ip;
+ $self->{_geoip_record} = $self->{_geoip_city}->record_by_addr($ip) or return;
+ return $self->{_geoip_record}->country_code;
+};
+
+sub set_country_name {
+ my $self = shift;
+ return $self->set_country_name_gc() if $self->{_geoip_city};
+ my $remote_ip = $self->qp->connection->remote_ip;
+ my $name = $self->{_geoip}->country_name_by_addr( $remote_ip ) or return;
+ $self->qp->connection->notes('geoip_country_name', $name);
+ return $name;
+};
+
+sub set_country_name_gc {
+ my $self = shift;
+ return if ! $self->{_geoip_record};
+ my $remote_ip = $self->qp->connection->remote_ip;
+ my $name = $self->{_geoip_record}->country_name() or return;
+ $self->qp->connection->notes('geoip_country_name', $name);
+ return $name;
+};
+
+sub set_continent {
+ my $self = shift;
+ return $self->set_continent_gc() if $self->{_geoip_city};
+ my $c_code = shift or return;
+ my $continent = $self->{_geoip}->continent_code_by_country_code( $c_code )
+ or return;
+ $self->qp->connection->notes('geoip_continent', $continent);
+ return $continent;
+};
+
+sub set_continent_gc {
+ my $self = shift;
+ return if ! $self->{_geoip_record};
+ my $continent = $self->{_geoip_record}->continent_code() or return;
+ $self->qp->connection->notes('geoip_continent', $continent);
+ return $continent;
+};
+
+sub set_distance_gc {
+ my $self = shift;
+ return if ! $self->{_geoip_record};
+
+ my ($self_lat, $self_lon) = $self->get_my_lat_lon() or return;
+ my ($sender_lat, $sender_lon) = $self->get_sender_lat_lon() or return;
+
+ eval 'use Math::Trig qw(great_circle_distance deg2rad)';
+ if ( $@ ) {
+ $self->log( LOGERROR, "can't calculate distance, Math::Trig not installed");
+ return;
+ };
+
+ # Notice the 90 - latitude: phi zero is at the North Pole.
+ sub NESW { deg2rad($_[0]), deg2rad(90 - $_[1]) };
+ my @me = NESW($self_lon, $self_lat );
+ my @sender = NESW($sender_lon, $sender_lat);
+ my $km = great_circle_distance(@me, @sender, 6378);
+ $km = sprintf("%.0f", $km);
+
+ $self->qp->connection->notes('geoip_distance', $km);
+ #$self->log( LOGINFO, "distance $km km");
+ return $km;
+};
+
+sub get_my_lat_lon {
+ my $self = shift;
+ return if ! $self->{_geoip_city};
+
+ if ( $self->{_latitude} && $self->{_longitude} ) {
+ return ( $self->{_latitude}, $self->{_longitude} ); # cached
+ };
+
+ my $ip = $self->{_args}{distance} or return;
+ my $record = $self->{_geoip_city}->record_by_addr($ip) or do {
+ $self->log( LOGERROR, "no record for my Geo::IP location");
+ return;
+ };
+
+ $self->{_latitude} = $record->latitude();
+ $self->{_longitude} = $record->longitude();
+
+ if ( ! $self->{_latitude} || ! $self->{_longitude} ) {
+ $self->log( LOGNOTICE, "could not get my lat/lon");
+ };
+ return ( $self->{_latitude}, $self->{_longitude} );
+};
+
+sub get_sender_lat_lon {
+ my $self = shift;
+
+ my $lat = $self->{_geoip_record}->latitude();
+ my $lon = $self->{_geoip_record}->longitude();
+ if ( ! $lat || ! $lon ) {
+ $self->log( LOGNOTICE, "could not get sender lat/lon");
+ return;
+ };
+ return ($lat, $lon);
+};
+
View
117 t/plugin_tests/ident/geoip
@@ -15,6 +15,12 @@ sub register_tests {
};
$self->register_test('test_geoip_lookup', 2);
+ $self->register_test('test_geoip_load_db', 2);
+ $self->register_test('test_geoip_init_cc', 2);
+ $self->register_test('test_set_country_code', 3);
+ $self->register_test('test_set_country_name', 3);
+ $self->register_test('test_set_continent', 3);
+ $self->register_test('test_set_distance', 3);
};
sub test_geoip_lookup {
@@ -26,4 +32,115 @@ sub test_geoip_lookup {
cmp_ok( $self->connection->notes('geoip_country'), 'eq', 'US', "note");
};
+sub test_geoip_load_db {
+ my $self = shift;
+
+ $self->open_geoip_db();
+
+ if ( $self->{_geoip_city} ) {
+ ok( ref $self->{_geoip_city}, "loaded GeoIP city db" );
+ }
+ else {
+ ok( "no GeoIP city db" );
+ };
+
+ if ( $self->{_geoip} ) {
+ ok( ref $self->{_geoip}, "loaded GeoIP db" );
+ }
+ else {
+ ok( "no GeoIP db" );
+ };
+};
+
+sub test_geoip_init_cc {
+ my $self = shift;
+
+ $self->{_my_country_code} = undef;
+ ok( ! $self->{_my_country_code}, "undefined");
+
+ my $test_ip = '208.175.177.10';
+ $self->{_args}{distance} = $test_ip;
+ $self->init_my_country_code( $test_ip );
+ cmp_ok( $self->{_my_country_code}, 'eq', 'US', "country set and matches");
+};
+
+sub test_set_country_code {
+ my $self = shift;
+
+ $self->qp->connection->remote_ip('');
+ my $cc = $self->set_country_code();
+ ok( ! $cc, "undef");
+
+ $self->qp->connection->remote_ip('24.24.24.24');
+ $cc = $self->set_country_code();
+ cmp_ok( $cc, 'eq', 'US', "$cc");
+
+ my $note = $self->connection->notes('geoip_country');
+ cmp_ok( $note, 'eq', 'US', "note has: $cc");
+};
+
+sub test_set_country_name {
+ my $self = shift;
+
+ $self->{_geoip_record} = undef;
+ $self->qp->connection->remote_ip('');
+ $self->set_country_code();
+ my $cn = $self->set_country_name();
+ ok( ! $cn, "undef") or warn "$cn\n";
+
+ $self->qp->connection->remote_ip('24.24.24.24');
+ $self->set_country_code();
+ $cn = $self->set_country_name();
+ cmp_ok( $cn, 'eq', 'United States', "$cn");
+
+ my $note = $self->connection->notes('geoip_country_name');
+ cmp_ok( $note, 'eq', 'United States', "note has: $cn");
+};
+
+sub test_set_continent {
+ my $self = shift;
+
+ $self->{_geoip_record} = undef;
+ $self->qp->connection->remote_ip('');
+ $self->set_country_code();
+ my $cn = $self->set_continent();
+ ok( ! $cn, "undef") or warn "$cn\n";
+
+ $self->qp->connection->remote_ip('24.24.24.24');
+ $self->set_country_code();
+ $cn = $self->set_continent() || '';
+ my $note = $self->connection->notes('geoip_continent');
+ if ( $cn ) {
+ cmp_ok( $cn, 'eq', 'NA', "$cn");
+ cmp_ok( $note, 'eq', 'NA', "note has: $cn");
+ }
+ else {
+ ok(1, "no continent data" );
+ ok(1, "no continent data" );
+ };
+};
+
+sub test_set_distance {
+ my $self = shift;
+
+ $self->{_geoip_record} = undef;
+ $self->qp->connection->remote_ip('');
+ $self->set_country_code();
+ my $cn = $self->set_distance_gc();
+ ok( ! $cn, "undef") or warn "$cn\n";
+
+ $self->qp->connection->remote_ip('24.24.24.24');
+ $self->set_country_code();
+ $cn = $self->set_distance_gc();
+ if ( $cn ) {
+ ok( $cn, "$cn km");
+
+ my $note = $self->connection->notes('geoip_distance');
+ ok( $note, "note has: $cn");
+ }
+ else {
+ ok( 1, "no distance data");
+ ok( 1, "no distance data");
+ }
+};
Something went wrong with that request. Please try again.