Skip to content
This repository

Add Maxmind support (using Maxmind Web Services) #227

Merged
merged 5 commits into from over 1 year ago

5 participants

Olivier Gonzalez Kevin Merritt Alex Reisner Daniel Pehrson Joe Marty
Olivier Gonzalez

Hi,

here is Maxmind support that I added 7 months ago on my fork.

I'm sorry to forgot to do a PR but it has the good point that this service has been used for 7 months now (on production servers)

Any comment is welcome and if you feel it okay, it would be great to have it merged into main repo.

Thanks.

Kevin Merritt

This is great!. Thanks. I will need to try maxmind because free geo ip is not very accurate.

Alex Reisner
Owner

Thanks for this. Just wanted to let you know that I will merge it, but need to re-think API key configuration first. Will probably be included in Geocoder 1.1.3.

Daniel Pehrson

This would be most excellent, I have a paid MaxMind GeoIP City database and would love to replace geoip+geocoder with just geocoder and a unified API.

Olivier Gonzalez

@alexreisner, good news !
@dpehrson, sorry but this pull request doesn't add support for downloaded database file. It only works with Maxmind GeoIP Web Services (http://www.maxmind.com/app/web_services).
I updated the title to be more explicit.

In fact, there are different levels of web services, which provides different amount of data. I did it for the "GeoIP City/ISP/Organization Web Service", additional stuff may be needed to have it fully functional for Services that provide more data.

Joe Marty
mltsy commented June 12, 2012

I would totally use this! :)

Alex Reisner alexreisner referenced this pull request June 15, 2012
Closed

IP address lookup #181

Alex Reisner alexreisner referenced this pull request July 06, 2012
Closed

geoip providers #255

Alex Reisner alexreisner merged commit 61a4ddb into from December 21, 2012
Alex Reisner alexreisner closed this December 21, 2012
Alex Reisner
Owner

Merged. Thanks!!

Alex Reisner
Owner

Could someone who uses MaxMind do me a huge favor and test this? I've asked MaxMind for some free requests but I'm not sure if/when they will grant them, and this is the last item blocking the release of gem version 1.1.6. To try it, just put this in your Gemfile:

gem 'geocoder', :git => 'git://github.com/alexreisner/geocoder.git'
Olivier Gonzalez

Hi Alex, I'm happy to see this is going forward!
It has been a while since I've used geocoder as I'm not anymore in charge of the project where I used it. Anyway I was too curious so I couldn't resist to launch a console :)

After bundling with master, I first got an exception when launching console:

undefined method `ip_lookup_api_key=' for Geocoder::Configuration:Class (NoMethodError)

So I took a look at the readme and searched for the other name you may have used but I saw you just removed that param. This isn't a good idea to me because of the following case :

Once you've configured your Maxmind API key in api_key attribute, you won't be able to use a text address lookup service (not ip) because it will use the same api_key.

IE:

> Geocoder.configure(:lookup => :google, :ip_lookup => :maxmind, :api_key => "my_maxmind_key")
=> {:timeout=>3, :lookup=>:google, :ip_lookup=>:maxmind, :language=>:en, :http_headers=>{}, :use_https=>false, :http_proxy=>nil, :https_proxy=>nil, :api_key=>"my_maxmind_key", :cache=>nil, :cache_prefix=>"geocoder:", :always_raise=>[], :units=>:mi, :distances=>:linear} 
> Geocoder.search("some.secret.ip.address")
=> [#<Geocoder::Result::Maxmind:0x007ff68ad3f468 @data=["US", "TX", "Houston", "77210", "29.xxxxx", "-95.xxxxxx", "618", "713", "Shell Information Technology International", "Shell Information Technology International"], @cache_hit=nil>] 
> Geocoder.search("Houston")
Google Geocoding API error: request denied.
=> []

It seems that Google doesn't like my Maxmind api_key :(

Or did I miss something?

Alex Reisner
Owner

@gonzoyumo thanks so much. Searching for an IP address should make it use MaxMind instead of Google:

Geocoder.search("74.200.247.59")

Let me know how that goes.

As for the configuration--you're absolutely right. That's why I didn't implement this sooner. I actually reworked the entire configuration syntax so you can specify keys, timeout, language, cache, etc for any number of APIs simultaneously. The intended use is something like this:

Geocoder.configure(
  :lookup => :yahoo,
  :ip_lookup => :maxmind,
  :yahoo => {:api_key => "...", :cache => Redis.new},
  :maxmind => {:api_key => "..."}
)

Any options specified at the top level will be defaults for all lookups. Make sense? This will all be added to the README and released, hopefully in the next couple of days.

Olivier Gonzalez

ahhh my bad, sorry :/ I should have looked at the code

Using correct configuration syntax it works like a charm :+1:

Just another remark: config/initializers/geocoder.rb sample file in the readme is ending with end instead of a parenthesis

Thanks you again for that great gem, I will definitely use it again when I'll need lookup service :)

Alex Reisner
Owner

Great. Thanks again. Really helpful.

One other thing, since it's about code you wrote: any thoughts on d97f399? Which documentation is your comment referring to? Ruby's CSV library? Would be great if we could get a sample response from MaxMind with some extended characters to use in a test.

Olivier Gonzalez

This was from Maxmind website, it seems that it has changed since but the documentation is still pretty clear about this:

All strings are returned in the ISO-8859-1 encoding. This encoding is also referred to as latin1.

you can find it in "ouput" section there: http://dev.maxmind.com/geoip/web-services

looking for a sample response...

Olivier Gonzalez

Here is a sample result string obtained with raw_data.inspect :

"FR,A3,Ch\xE2lo-saint-mars,,48.427601,2.065400,0,0,\"Free SAS\",\"Free SAS\""

As you may see, maxmind documentation provide a ruby 1.9 example and they use:

response.body.encode('utf-8', 'iso-8859-1').parse_csv

.encode('utf-8', 'iso-8859-1').parse_csv works well when applied to raw_data string, so you may use it I think

I don't know about ruby 1.8 :/

Alex Reisner
Owner

Thanks for the sample data! Unfortunately neither the current solution nor theirs seems to work!

I wonder if Geocoder should be doing any conversion at all. Granted, among us westerners UTF-8 seems to be the current encoding of choice, but some people might want to convert to something else (eg: UTF-16 or -32), in which case this would be some wasted CPU cycles.

I do think it's worthwhile, though. Most of the other supported services (including Google and Bing) return UTF-8 and perhaps more care should be given to ensure that all results are UTF-8 encoded.

However, I am not going to do the conversion with Ruby 1.8.x. Unless I'm mistaken, every method of changing character encoding in Ruby 1.8.x requires some external system dependency (like iconv), which is too much complexity for Geocoder. I don't like this inconsistency between Ruby versions, and will make a note of it, but I'm not going to introduce a large dependency for what is sort of a minor issue.

Alex Reisner
Owner

Also, what's the reasoning behind the substrings in Geocoder::Result::Maxmind#isp and #organization? Those values appear to be getting truncated.

Olivier Gonzalez

hmm, I didn't catch that. To be honest I don't remember why I did such a thing (I should have written a comment), but I think it was mandatory to get clean values ^^

isp and organization are the only attribute that are wrapped with double quotes in the result string. I suppose there was something weird with this and I was forced to manually trim them.

It seems that it's not the case anymore so it should be dropped as they are indeed cutting values now.

Olivier Gonzalez

Ah, one more thing about my code: I only focused on supporting the "City/ISP/Org" service (the one we were using).

But Maxmind also provides other services which contain less or more attributes, making the current code failing for them.

This is due to the fact that Maxmind response length is different for each services, see http://dev.maxmind.com/geoip/web-services#Output_field_order-3

So it's worth warning user about this or implementing other services for full support.

thanks

Alex Reisner
Owner

Thanks for pointing out the different response types. Sounds like we need to support all four types. Here's what I'm going to do:

  1. Fix the isp/organization trimming.
  2. Implement configuration/support for the four MaxMind services/response types.
  3. Remove string encoding conversion.

The string encoding is too complicated to implement correctly/consistently before the 1.1.6 release (which I'd like to do today or tomorrow). I'll create an issue about handling encoding from all APIs consistently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
16  README.rdoc
Source Rendered
@@ -235,6 +235,12 @@ By default Geocoder uses Google's geocoding API to fetch coordinates and street
235 235
   # to use an API key:
236 236
   Geocoder::Configuration.api_key = "..."
237 237
 
  238
+  # IP geocoding service :
  239
+  Geocoder::Configuration.ip_lookup = :maxmind
  240
+
  241
+  # to use an API key for IP geocoding service:
  242
+  Geocoder::Configuration.ip_lookup_api_key = "..."
  243
+
238 244
   # geocoding service request timeout, in seconds (default 3):
239 245
   Geocoder::Configuration.timeout = 5
240 246
 
@@ -339,6 +345,16 @@ Documentation:: http://github.com/fiorix/freegeoip/blob/master/README.rst
339 345
 Terms of Service:: ?
340 346
 Limitations:: ?
341 347
 
  348
+==== MaxMind City
  349
+
  350
+API key:: required
  351
+Quota:: Requests Packs can be purchased (starting at 50,000 for 20$)
  352
+Region:: world
  353
+SSL support:: no
  354
+Languages:: English
  355
+Documentation:: http://www.maxmind.com/app/web_services
  356
+Terms of Service:: ?
  357
+Limitations:: ?
342 358
 
343 359
 == Caching
344 360
 
4  lib/geocoder.rb
@@ -64,7 +64,7 @@ def street_lookups
64 64
   # All IP address lookups, default first.
65 65
   #
66 66
   def ip_lookups
67  
-    [:freegeoip]
  67
+    [:freegeoip, :maxmind]
68 68
   end
69 69
 
70 70
 
@@ -77,7 +77,7 @@ def ip_lookups
77 77
   #
78 78
   def lookup(query)
79 79
     if ip_address?(query)
80  
-      get_lookup(ip_lookups.first)
  80
+      get_lookup(Configuration.ip_lookup || ip_lookups.first)
81 81
     else
82 82
       get_lookup(Configuration.lookup || street_lookups.first)
83 83
     end
8  lib/geocoder/configuration.rb
@@ -28,6 +28,14 @@ def self.options_and_defaults
28 28
         # for Google Premier use a 3-element array: [key, client, channel]
29 29
         [:api_key, nil],
30 30
 
  31
+        # name of IP geocoding service (symbol)
  32
+        # FIXME: temporary added for Maxmind support, need clean rewrite
  33
+        [:ip_lookup, nil],
  34
+
  35
+        # API key for IP geocoding service
  36
+        # FIXME: temporary added for Maxmind support, need clean rewrite
  37
+        [:ip_lookup_api_key, nil],
  38
+
31 39
         # cache object (must respond to #[], #[]=, and #keys)
32 40
         [:cache, nil],
33 41
 
41  lib/geocoder/lookups/maxmind.rb
... ...
@@ -0,0 +1,41 @@
  1
+require 'geocoder/lookups/base'
  2
+require 'geocoder/results/maxmind'
  3
+require 'csv'
  4
+
  5
+module Geocoder::Lookup
  6
+  class Maxmind < Base
  7
+
  8
+    private # ---------------------------------------------------------------
  9
+
  10
+    def results(query, reverse = false)
  11
+      # don't look up a loopback address, just return the stored result
  12
+      return [reserved_result] if loopback_address?(query)
  13
+      begin
  14
+        doc = fetch_data(query, reverse)
  15
+        if doc && doc.size == 10
  16
+          return [doc]
  17
+        else
  18
+          warn "Maxmind error : #{doc[10]}" if doc
  19
+          return []
  20
+        end
  21
+      rescue StandardError => err
  22
+        raise_error(err)
  23
+        return []
  24
+      end
  25
+    end
  26
+
  27
+    def parse_raw_data(raw_data)
  28
+      # Maxmind just returns text/plain as csv format but according to documentation,
  29
+      # we get ISO-8859-1 encoded string. We need to convert it.
  30
+      CSV.parse_line raw_data.force_encoding("ISO-8859-1").encode("UTF-8")
  31
+    end
  32
+
  33
+    def reserved_result
  34
+      ",,,,0,0,0,0,,"
  35
+    end
  36
+
  37
+    def query_url(query, reverse = false)
  38
+      "http://geoip3.maxmind.com/f?l=#{Geocoder::Configuration.ip_lookup_api_key}&i=#{query}"
  39
+    end
  40
+  end
  41
+end
55  lib/geocoder/results/maxmind.rb
... ...
@@ -0,0 +1,55 @@
  1
+require 'geocoder/results/base'
  2
+
  3
+module Geocoder::Result
  4
+  class Maxmind < Base
  5
+
  6
+    def address(format = :full)
  7
+      s = state_code.to_s == "" ? "" : ", #{state_code}"
  8
+      "#{city}#{s} #{postal_code}, #{country_code}".sub(/^[ ,]*/, "")
  9
+    end
  10
+
  11
+    def country_code
  12
+      @data[0]
  13
+    end
  14
+
  15
+    def state_code
  16
+      @data[1]
  17
+    end
  18
+
  19
+    def city
  20
+      @data[2]
  21
+    end
  22
+
  23
+    def postal_code
  24
+      @data[3]
  25
+    end
  26
+
  27
+    def coordinates
  28
+      [@data[4].to_f, @data[5].to_f]
  29
+    end
  30
+
  31
+    def metrocode
  32
+      @data[6]
  33
+    end
  34
+
  35
+    def area_code
  36
+      @data[7]
  37
+    end
  38
+
  39
+    def isp
  40
+      @data[8][1,@data[8].length-2]
  41
+    end
  42
+
  43
+    def organization
  44
+      @data[9][1,@data[9].length-2]
  45
+    end
  46
+
  47
+    def country #not given by MaxMind
  48
+      country_code
  49
+    end
  50
+
  51
+    def state #not given by MaxMind
  52
+      state_code
  53
+    end
  54
+  end
  55
+end
1  test/fixtures/maxmind_74_200_247_59.txt
... ...
@@ -0,0 +1 @@
  1
+US,TX,Plano,75093,33.034698,-96.813400,623,972,"Layered Technologies , US","Layered Technologies , US"
1  test/fixtures/maxmind_no_results.txt
... ...
@@ -0,0 +1 @@
  1
+,,,,,,,,,,IP_NOT_FOUND
6  test/lookup_test.rb
@@ -27,4 +27,10 @@ def test_yahoo_app_id
27 27
     g = Geocoder::Lookup::Yahoo.new
28 28
     assert_match "appid=MY_KEY", g.send(:query_url, "Madison Square Garden, New York, NY  10001, United States")
29 29
   end
  30
+
  31
+  def test_maxmind_api_key
  32
+    Geocoder::Configuration.ip_lookup_api_key = "MY_KEY"
  33
+    g = Geocoder::Lookup::Maxmind.new
  34
+    assert_match "l=MY_KEY", g.send(:query_url, "74.200.247.59")
  35
+  end
30 36
 end
14  test/services_test.rb
@@ -94,6 +94,20 @@ def test_freegeoip_result_components
94 94
     assert_equal "Plano, TX 75093, United States", result.address
95 95
   end
96 96
 
  97
+  # --- MaxMind ---
  98
+
  99
+  def test_maxmind_result_on_ip_address_search
  100
+    Geocoder::Configuration.ip_lookup = :maxmind
  101
+    result = Geocoder.search("74.200.247.59").first
  102
+    assert result.is_a?(Geocoder::Result::Maxmind)
  103
+  end
  104
+
  105
+  def test_maxmind_result_components
  106
+    Geocoder::Configuration.ip_lookup = :maxmind
  107
+    result = Geocoder.search("74.200.247.59").first
  108
+    assert_equal "Plano, TX 75093, US", result.address
  109
+  end
  110
+
97 111
 
98 112
   # --- Bing ---
99 113
 
13  test/test_helper.rb
@@ -133,6 +133,19 @@ def fetch_raw_data(query, reverse = false)
133 133
       end
134 134
     end
135 135
 
  136
+    class Maxmind < Base
  137
+      private #-----------------------------------------------------------------
  138
+      def fetch_raw_data(query, reverse = false)
  139
+        raise TimeoutError if query == "timeout"
  140
+        raise SocketError if query == "socket_error"
  141
+        file = case query
  142
+          when "no results";  :no_results
  143
+          else                "74_200_247_59"
  144
+        end
  145
+        read_fixture "maxmind_#{file}.txt"
  146
+      end
  147
+    end
  148
+
136 149
     class Bing < Base
137 150
       private #-----------------------------------------------------------------
138 151
       def fetch_raw_data(query, reverse = false)
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.