Skip to content
This repository

Geoip #28

Closed
wants to merge 2 commits into from

1 participant

Matt Simerson
Matt Simerson
Collaborator

added GeoIP City support, continent, distance

wrote lots of succinct POD, and lots more tests.

This plugin saves geographic information in the following connection
notes:

     geoip_country      - 2 char country code
     geoip_country_name - full english name of country
     geoip_continent    - 2 char continent code
     geoip_distance     - distance in kilometers

   And adds entries like this to your logs:

     (connect) ident::geoip: US, United States, NA,    1319 km
     (connect) ident::geoip: IN, India, AS,    13862 km
     (connect) ident::geoip: fail: no results
     (connect) ident::geoip: CA, Canada, NA,   2464 km
Matt Simerson msimerson closed this August 05, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Jun 06, 2012
Matt Simerson geoip: added geodesic distance calculations
which require the GeoIP city data and Math::Complex
cf85811
Matt Simerson geoip: added tests and a couple fixes c7824e1
This page is out of date. Refresh to see the latest.
287  plugins/ident/geoip
... ...
@@ -1,17 +1,102 @@
1 1
 #!perl -w
2 2
 
  3
+=head1 NAME
  4
+
  5
+geoip - provide geographic information about mail senders.
  6
+
3 7
 =head1 SYNOPSIS
4 8
 
5  
-This plugin uses MaxMind's GeoIP service and the Geo::IP perl module to
6  
-do a lookup on incoming connections and record the country of origin.
  9
+Use MaxMind's GeoIP databases and the Geo::IP perl module to report geographic
  10
+information about incoming connections.
  11
+
  12
+=head1 DESCRIPTION
  13
+
  14
+This plugin saves geographic information in the following connection notes:
  15
+
  16
+  geoip_country      - 2 char country code
  17
+  geoip_country_name - full english name of country
  18
+  geoip_continent    - 2 char continent code
  19
+  geoip_distance     - distance in kilometers
  20
+
  21
+And adds entries like this to your logs:
  22
+
  23
+  (connect) ident::geoip: US, United States, NA,    1319 km
  24
+  (connect) ident::geoip: IN, India, AS,    13862 km
  25
+  (connect) ident::geoip: fail: no results
  26
+  (connect) ident::geoip: CA, Canada, NA,   2464 km
  27
+  (connect) ident::geoip: US, United States, NA,    2318 km
  28
+  (connect) ident::geoip: PK, Pakistan, AS,    12578 km
  29
+  (connect) ident::geoip: TJ, Tajikistan, AS,  11965 km
  30
+  (connect) ident::geoip: AT, Austria, EU,     8745 km
  31
+  (connect) ident::geoip: IR, Iran, Islamic Republic of, AS,   12180 km
  32
+  (connect) ident::geoip: BY, Belarus, EU,     9030 km
  33
+  (connect) ident::geoip: CN, China, AS,   11254 km
  34
+  (connect) ident::geoip: PA, Panama, NA,  3163 km
  35
+
  36
+Calculating the distance has three prerequsites:
  37
+
  38
+  1. The MaxMind city database (free or subscription)
  39
+  2. The Math::Complex perl module
  40
+  3. The IP address of this mail server (see CONFIG)
  41
+
  42
+Other plugins can utilize the geographic notes to alter the
  43
+connection, reject, greylist, etc.
  44
+
  45
+=head1 CONFIG
  46
+
  47
+The following options can be appended in this plugins config/plugins entry.
  48
+
  49
+=head2 distance <IP Address>
  50
+
  51
+Enables geodesic distance calculation. Will calculate the distance "as the
  52
+crow flies" from the remote mail server. Accepts a single argument, the IP
  53
+address to calculate the distance from. This will typically be the public
  54
+IP of your mail server.
  55
+
  56
+  ident/geoip [ distance 192.0.1.5 ]
  57
+
  58
+Default: none. (no distance calculations)
  59
+
  60
+=head2 db_dir </path/to/GeoIP>
  61
+
  62
+The path to the GeoIP database directory.
  63
+
  64
+  ident/geoip [ db_dir /etc/GeoIP ]
  65
+
  66
+Default: /usr/local/share/GeoIP
  67
+
  68
+=head1 LIMITATIONS
  69
+
  70
+The distance calculations are more concerned with being fast than accurate.
  71
+The MaxMind location data is collected from whois and is of limited accuracy.
  72
+MaxMind offers more accurate data for a fee.
  73
+
  74
+For distance calculations, the earth is considered a perfect sphere. In
  75
+reality, it is not. Accuracy should be within 1%.
  76
+
  77
+This plugin does not update the GeoIP databases. You may want to.
  78
+
  79
+=head1 CHANGES
  80
+
  81
+2012-06 - Matt Simerson - added GeoIP City support, continent, distance
7 82
 
8  
-Thats all it does.
  83
+2012-05 - Matt Simerson - added geoip_country_name note, added tests
9 84
 
10  
-It logs the 2 char country code to connection note I<geoip_country>.
11  
-It logs the country name to the connection note I<geoip_country_name>.
  85
+=head1 SEE ALSO
12 86
 
13  
-Other plugins can use that info to do things to the connection, like
14  
-reject or greylist.
  87
+MaxMind: http://www.maxmind.com/
  88
+
  89
+Databases: http://geolite.maxmind.com/download/geoip/database
  90
+
  91
+It may become worth adding support for Geo::IPfree, which uses another
  92
+data source: http://software77.net/geo-ip/
  93
+
  94
+=head1 ACKNOWLEDGEMENTS
  95
+
  96
+Stevan Bajic, the DSPAM author, who suggested SNARE, which describes using
  97
+geodesic distance to determine spam probability. The research paper on SNARE
  98
+can be found here:
  99
+http://smartech.gatech.edu/bitstream/handle/1853/25135/GT-CSE-08-02.pdf
15 100
 
16 101
 =cut
17 102
 
@@ -19,10 +104,16 @@ use strict;
19 104
 use warnings;
20 105
 
21 106
 use Qpsmtpd::Constants;
22  
-#use Geo::IP;  # eval'ed in register()
  107
+#use Geo::IP;     # eval'ed in register()
  108
+#use Math::Trig;  # eval'ed in set_distance_gc
23 109
 
24 110
 sub register {
25  
-    my $self = shift;
  111
+    my ($self, $qp ) = shift, shift;
  112
+
  113
+    $self->log(LOGERROR, "Bad arguments") if @_ % 2;
  114
+    $self->{_args} = { @_ };
  115
+    $self->{_args}{db_dir}   ||= '/usr/local/share/GeoIP';
  116
+
26 117
     eval 'use Geo::IP';
27 118
     if ( $@ ) {
28 119
         warn "could not load Geo::IP";
@@ -30,30 +121,192 @@ sub register {
30 121
         return;
31 122
     };
32 123
 
  124
+# Note that opening the GeoIP DB only in register has caused problems before:
  125
+# https://github.com/smtpd/qpsmtpd/commit/29ea9516806e9a8ca6519fcf987dbd684793ebdd#plugins/ident/geoip
  126
+# Opening the DB anew for every connection is horribly inefficient.
  127
+# Instead, attempt to reopen upon connect if the DB connection fails.
  128
+    $self->open_geoip_db();
  129
+
  130
+    $self->init_my_country_code();
  131
+
33 132
     $self->register_hook( 'connect', 'connect_handler' );
34 133
 };
35 134
 
36 135
 sub connect_handler {
37 136
     my $self = shift;
38 137
 
39  
-    my $geoip = Geo::IP->new();
40  
-    my $remote_ip = $self->qp->connection->remote_ip;
  138
+    # reopen the DB if Geo::IP failed due to DB update
  139
+    $self->open_geoip_db();
41 140
 
42  
-    my $c_code = $geoip->country_code_by_addr( $remote_ip ) or do {
  141
+    my $c_code = $self->set_country_code() or do {
43 142
         $self->log( LOGINFO, "fail: no results" );
44 143
         return DECLINED;
45 144
     };
  145
+    $self->qp->connection->notes('geoip_country', $c_code);
46 146
 
47  
-    my $c_name = $geoip->country_name_by_addr( $remote_ip );
48  
-    if ( $c_name ) {
49  
-        $self->connection->notes('geoip_country_name', $c_name);
50  
-    };
  147
+    my $c_name = $self->set_country_name();
  148
+    my ($continent_code, $distance);
51 149
 
52  
-    $self->connection->notes('geoip_country', $c_code);
  150
+    if ( $self->{_my_country_code} ) {
  151
+        $continent_code = $self->set_continent( $c_code );
  152
+        $distance = $self->set_distance_gc();
  153
+    };
53 154
 
54 155
     my $message  = $c_code;
55 156
        $message .= ", $c_name" if $c_name;
  157
+       $message .= ", $continent_code" if $continent_code && $continent_code ne '--';
  158
+       $message .= ", \t$distance km" if $distance;
56 159
     $self->log(LOGINFO, $message);
57 160
 
58 161
     return DECLINED;
59 162
 }
  163
+
  164
+sub open_geoip_db {
  165
+    my $self = shift;
  166
+
  167
+    # this might detect if the DB connection failed. If not, this is where
  168
+    # to add more code to do it.
  169
+    return if ( defined $self->{_geoip_city} || defined $self->{_geoip} );
  170
+
  171
+    # The methods for using GeoIP work differently for the City vs Country DB
  172
+    # save the handles in different locations
  173
+    my $db_dir = $self->{_args}{db_dir};
  174
+    foreach my $db ( qw/ GeoIPCity GeoLiteCity / ) {
  175
+        if ( -f "$db_dir/$db.dat" ) {
  176
+            $self->log(LOGDEBUG, "using db $db");
  177
+            $self->{_geoip_city} = Geo::IP->open( "$db_dir/$db.dat" );
  178
+        }
  179
+    };
  180
+
  181
+    # can't think of a good reason to load country if city data is present
  182
+    if ( ! $self->{_geoip_city} ) {
  183
+        $self->log(LOGDEBUG, "using default db");
  184
+        $self->{_geoip} = Geo::IP->new();  # loads default Country DB
  185
+    };
  186
+};
  187
+
  188
+sub init_my_country_code {
  189
+    my $self = shift;
  190
+    my $ip = $self->{_args}{distance} or return;
  191
+    $self->{_my_country_code} = $self->get_country_code( $ip );
  192
+};
  193
+
  194
+sub set_country_code {
  195
+    my $self = shift;
  196
+    return $self->get_country_code_gc() if $self->{_geoip_city};
  197
+    my $remote_ip = $self->qp->connection->remote_ip;
  198
+    my $code = $self->get_country_code();
  199
+    $self->qp->connection->notes('geoip_country', $code);
  200
+    return $code;
  201
+};
  202
+
  203
+sub get_country_code {
  204
+    my $self = shift;
  205
+    my $ip = shift || $self->qp->connection->remote_ip;
  206
+    return $self->get_country_code_gc( $ip ) if $self->{_geoip_city};
  207
+    return $self->{_geoip}->country_code_by_addr( $ip );
  208
+};
  209
+
  210
+sub get_country_code_gc {
  211
+    my $self = shift;
  212
+    my $ip   = shift || $self->qp->connection->remote_ip;
  213
+    $self->{_geoip_record} = $self->{_geoip_city}->record_by_addr($ip) or return;
  214
+    return $self->{_geoip_record}->country_code;
  215
+};
  216
+
  217
+sub set_country_name {
  218
+    my $self = shift;
  219
+    return $self->set_country_name_gc() if $self->{_geoip_city};
  220
+    my $remote_ip = $self->qp->connection->remote_ip;
  221
+    my $name = $self->{_geoip}->country_name_by_addr( $remote_ip ) or return;
  222
+    $self->qp->connection->notes('geoip_country_name', $name);
  223
+    return $name;
  224
+};
  225
+
  226
+sub set_country_name_gc {
  227
+    my $self = shift;
  228
+    return if ! $self->{_geoip_record};
  229
+    my $remote_ip = $self->qp->connection->remote_ip;
  230
+    my $name = $self->{_geoip_record}->country_name() or return;
  231
+    $self->qp->connection->notes('geoip_country_name', $name);
  232
+    return $name;
  233
+};
  234
+
  235
+sub set_continent {
  236
+    my $self = shift;
  237
+    return $self->set_continent_gc() if $self->{_geoip_city};
  238
+    my $c_code = shift or return;
  239
+    my $continent = $self->{_geoip}->continent_code_by_country_code( $c_code )
  240
+        or return;
  241
+    $self->qp->connection->notes('geoip_continent', $continent);
  242
+    return $continent;
  243
+};
  244
+
  245
+sub set_continent_gc {
  246
+    my $self = shift;
  247
+    return if ! $self->{_geoip_record};
  248
+    my $continent = $self->{_geoip_record}->continent_code() or return;
  249
+    $self->qp->connection->notes('geoip_continent', $continent);
  250
+    return $continent;
  251
+};
  252
+
  253
+sub set_distance_gc {
  254
+    my $self = shift;
  255
+    return if ! $self->{_geoip_record};
  256
+
  257
+    my ($self_lat, $self_lon) = $self->get_my_lat_lon() or return;
  258
+    my ($sender_lat, $sender_lon) = $self->get_sender_lat_lon() or return;
  259
+
  260
+    eval 'use Math::Trig qw(great_circle_distance deg2rad)';
  261
+    if ( $@ ) {
  262
+        $self->log( LOGERROR, "can't calculate distance, Math::Trig not installed");
  263
+        return;
  264
+    };
  265
+
  266
+    # Notice the 90 - latitude: phi zero is at the North Pole.
  267
+    sub NESW { deg2rad($_[0]), deg2rad(90 - $_[1]) };
  268
+    my @me     = NESW($self_lon, $self_lat );
  269
+    my @sender = NESW($sender_lon, $sender_lat);
  270
+    my $km     = great_circle_distance(@me, @sender, 6378);
  271
+    $km = sprintf("%.0f", $km);
  272
+
  273
+    $self->qp->connection->notes('geoip_distance', $km);
  274
+    #$self->log( LOGINFO, "distance $km km");
  275
+    return $km;
  276
+};
  277
+
  278
+sub get_my_lat_lon {
  279
+    my $self = shift;
  280
+    return if ! $self->{_geoip_city};
  281
+
  282
+    if ( $self->{_latitude} && $self->{_longitude} ) {
  283
+        return ( $self->{_latitude}, $self->{_longitude} ); # cached
  284
+    };
  285
+
  286
+    my $ip     = $self->{_args}{distance} or return;
  287
+    my $record = $self->{_geoip_city}->record_by_addr($ip) or do {
  288
+        $self->log( LOGERROR, "no record for my Geo::IP location");
  289
+        return;
  290
+    };
  291
+
  292
+    $self->{_latitude}  = $record->latitude();
  293
+    $self->{_longitude} = $record->longitude();
  294
+
  295
+    if ( ! $self->{_latitude} || ! $self->{_longitude} ) {
  296
+        $self->log( LOGNOTICE, "could not get my lat/lon");
  297
+    };
  298
+    return ( $self->{_latitude}, $self->{_longitude} );
  299
+};
  300
+
  301
+sub get_sender_lat_lon {
  302
+    my $self = shift;
  303
+
  304
+    my $lat = $self->{_geoip_record}->latitude();
  305
+    my $lon = $self->{_geoip_record}->longitude();
  306
+    if ( ! $lat || ! $lon ) {
  307
+        $self->log( LOGNOTICE, "could not get sender lat/lon");
  308
+        return;
  309
+    };
  310
+    return ($lat, $lon);
  311
+};
  312
+
117  t/plugin_tests/ident/geoip
@@ -15,6 +15,12 @@ sub register_tests {
15 15
     };
16 16
 
17 17
     $self->register_test('test_geoip_lookup', 2);
  18
+    $self->register_test('test_geoip_load_db', 2);
  19
+    $self->register_test('test_geoip_init_cc', 2);
  20
+    $self->register_test('test_set_country_code', 3);
  21
+    $self->register_test('test_set_country_name', 3);
  22
+    $self->register_test('test_set_continent', 3);
  23
+    $self->register_test('test_set_distance', 3);
18 24
 };
19 25
 
20 26
 sub test_geoip_lookup {
@@ -26,4 +32,115 @@ sub test_geoip_lookup {
26 32
     cmp_ok( $self->connection->notes('geoip_country'), 'eq', 'US', "note");
27 33
 };
28 34
 
  35
+sub test_geoip_load_db {
  36
+    my $self = shift;
  37
+
  38
+    $self->open_geoip_db();
  39
+
  40
+    if ( $self->{_geoip_city} ) {
  41
+        ok( ref $self->{_geoip_city}, "loaded GeoIP city db" );
  42
+    }
  43
+    else {
  44
+        ok( "no GeoIP city db" );
  45
+    };
  46
+
  47
+    if ( $self->{_geoip} ) {
  48
+        ok( ref $self->{_geoip}, "loaded GeoIP db" );
  49
+    }
  50
+    else {
  51
+        ok( "no GeoIP db" );
  52
+    };
  53
+};
  54
+
  55
+sub test_geoip_init_cc {
  56
+    my $self = shift;
  57
+
  58
+    $self->{_my_country_code} = undef;
  59
+    ok( ! $self->{_my_country_code}, "undefined");
  60
+
  61
+    my $test_ip = '208.175.177.10';
  62
+    $self->{_args}{distance} = $test_ip;
  63
+    $self->init_my_country_code( $test_ip );
  64
+    cmp_ok( $self->{_my_country_code}, 'eq', 'US', "country set and matches");
  65
+};
  66
+
  67
+sub test_set_country_code {
  68
+    my $self = shift;
  69
+
  70
+    $self->qp->connection->remote_ip('');
  71
+    my $cc = $self->set_country_code();
  72
+    ok( ! $cc, "undef");
  73
+
  74
+    $self->qp->connection->remote_ip('24.24.24.24');
  75
+    $cc = $self->set_country_code();
  76
+    cmp_ok( $cc, 'eq', 'US', "$cc");
  77
+
  78
+    my $note = $self->connection->notes('geoip_country');
  79
+    cmp_ok( $note, 'eq', 'US', "note has: $cc");
  80
+};
  81
+
  82
+sub test_set_country_name {
  83
+    my $self = shift;
  84
+
  85
+    $self->{_geoip_record} = undef;
  86
+    $self->qp->connection->remote_ip('');
  87
+    $self->set_country_code();
  88
+    my $cn = $self->set_country_name();
  89
+    ok( ! $cn, "undef") or warn "$cn\n";
  90
+
  91
+    $self->qp->connection->remote_ip('24.24.24.24');
  92
+    $self->set_country_code();
  93
+    $cn = $self->set_country_name();
  94
+    cmp_ok( $cn, 'eq', 'United States', "$cn");
  95
+
  96
+    my $note = $self->connection->notes('geoip_country_name');
  97
+    cmp_ok( $note, 'eq', 'United States', "note has: $cn");
  98
+};
  99
+
  100
+sub test_set_continent {
  101
+    my $self = shift;
  102
+
  103
+    $self->{_geoip_record} = undef;
  104
+    $self->qp->connection->remote_ip('');
  105
+    $self->set_country_code();
  106
+    my $cn = $self->set_continent();
  107
+    ok( ! $cn, "undef") or warn "$cn\n";
  108
+
  109
+    $self->qp->connection->remote_ip('24.24.24.24');
  110
+    $self->set_country_code();
  111
+    $cn = $self->set_continent() || '';
  112
+    my $note = $self->connection->notes('geoip_continent');
  113
+    if ( $cn ) {
  114
+        cmp_ok( $cn, 'eq', 'NA', "$cn");
  115
+        cmp_ok( $note, 'eq', 'NA', "note has: $cn");
  116
+    }
  117
+    else {
  118
+        ok(1, "no continent data" );
  119
+        ok(1, "no continent data" );
  120
+    };
  121
+};
  122
+
  123
+sub test_set_distance {
  124
+    my $self = shift;
  125
+
  126
+    $self->{_geoip_record} = undef;
  127
+    $self->qp->connection->remote_ip('');
  128
+    $self->set_country_code();
  129
+    my $cn = $self->set_distance_gc();
  130
+    ok( ! $cn, "undef") or warn "$cn\n";
  131
+
  132
+    $self->qp->connection->remote_ip('24.24.24.24');
  133
+    $self->set_country_code();
  134
+    $cn = $self->set_distance_gc();
  135
+    if ( $cn ) {
  136
+        ok( $cn, "$cn km");
  137
+
  138
+        my $note = $self->connection->notes('geoip_distance');
  139
+        ok( $note, "note has: $cn");
  140
+    }
  141
+    else {
  142
+        ok( 1, "no distance data");
  143
+        ok( 1, "no distance data");
  144
+    }
  145
+};
29 146
 
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.