Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add Maxmind support (using Maxmind Web Services) #227

Merged
merged 5 commits into from

5 participants

Olivier Gonzalez Kevin Merritt Alex Reisner Daniel Pehrson Joe Marty
Olivier Gonzalez

Hi,

here is Maxmind support that I added 7 months ago on my fork.

I'm sorry to forgot to do a PR but it has the good point that this service has been used for 7 months now (on production servers)

Any comment is welcome and if you feel it okay, it would be great to have it merged into main repo.

Thanks.

Kevin Merritt

This is great!. Thanks. I will need to try maxmind because free geo ip is not very accurate.

Alex Reisner
Owner

Thanks for this. Just wanted to let you know that I will merge it, but need to re-think API key configuration first. Will probably be included in Geocoder 1.1.3.

Daniel Pehrson

This would be most excellent, I have a paid MaxMind GeoIP City database and would love to replace geoip+geocoder with just geocoder and a unified API.

Olivier Gonzalez

@alexreisner, good news !
@dpehrson, sorry but this pull request doesn't add support for downloaded database file. It only works with Maxmind GeoIP Web Services (http://www.maxmind.com/app/web_services).
I updated the title to be more explicit.

In fact, there are different levels of web services, which provides different amount of data. I did it for the "GeoIP City/ISP/Organization Web Service", additional stuff may be needed to have it fully functional for Services that provide more data.

Joe Marty

I would totally use this! :)

Alex Reisner alexreisner merged commit 61a4ddb into from
Alex Reisner
Owner

Merged. Thanks!!

Alex Reisner
Owner

Could someone who uses MaxMind do me a huge favor and test this? I've asked MaxMind for some free requests but I'm not sure if/when they will grant them, and this is the last item blocking the release of gem version 1.1.6. To try it, just put this in your Gemfile:

gem 'geocoder', :git => 'git://github.com/alexreisner/geocoder.git'
Olivier Gonzalez

Hi Alex, I'm happy to see this is going forward!
It has been a while since I've used geocoder as I'm not anymore in charge of the project where I used it. Anyway I was too curious so I couldn't resist to launch a console :)

After bundling with master, I first got an exception when launching console:

undefined method `ip_lookup_api_key=' for Geocoder::Configuration:Class (NoMethodError)

So I took a look at the readme and searched for the other name you may have used but I saw you just removed that param. This isn't a good idea to me because of the following case :

Once you've configured your Maxmind API key in api_key attribute, you won't be able to use a text address lookup service (not ip) because it will use the same api_key.

IE:

> Geocoder.configure(:lookup => :google, :ip_lookup => :maxmind, :api_key => "my_maxmind_key")
=> {:timeout=>3, :lookup=>:google, :ip_lookup=>:maxmind, :language=>:en, :http_headers=>{}, :use_https=>false, :http_proxy=>nil, :https_proxy=>nil, :api_key=>"my_maxmind_key", :cache=>nil, :cache_prefix=>"geocoder:", :always_raise=>[], :units=>:mi, :distances=>:linear} 
> Geocoder.search("some.secret.ip.address")
=> [#<Geocoder::Result::Maxmind:0x007ff68ad3f468 @data=["US", "TX", "Houston", "77210", "29.xxxxx", "-95.xxxxxx", "618", "713", "Shell Information Technology International", "Shell Information Technology International"], @cache_hit=nil>] 
> Geocoder.search("Houston")
Google Geocoding API error: request denied.
=> []

It seems that Google doesn't like my Maxmind api_key :(

Or did I miss something?

Alex Reisner
Owner

@gonzoyumo thanks so much. Searching for an IP address should make it use MaxMind instead of Google:

Geocoder.search("74.200.247.59")

Let me know how that goes.

As for the configuration--you're absolutely right. That's why I didn't implement this sooner. I actually reworked the entire configuration syntax so you can specify keys, timeout, language, cache, etc for any number of APIs simultaneously. The intended use is something like this:

Geocoder.configure(
  :lookup => :yahoo,
  :ip_lookup => :maxmind,
  :yahoo => {:api_key => "...", :cache => Redis.new},
  :maxmind => {:api_key => "..."}
)

Any options specified at the top level will be defaults for all lookups. Make sense? This will all be added to the README and released, hopefully in the next couple of days.

Olivier Gonzalez

ahhh my bad, sorry :/ I should have looked at the code

Using correct configuration syntax it works like a charm :+1:

Just another remark: config/initializers/geocoder.rb sample file in the readme is ending with end instead of a parenthesis

Thanks you again for that great gem, I will definitely use it again when I'll need lookup service :)

Alex Reisner
Owner

Great. Thanks again. Really helpful.

One other thing, since it's about code you wrote: any thoughts on d97f399? Which documentation is your comment referring to? Ruby's CSV library? Would be great if we could get a sample response from MaxMind with some extended characters to use in a test.

Olivier Gonzalez

This was from Maxmind website, it seems that it has changed since but the documentation is still pretty clear about this:

All strings are returned in the ISO-8859-1 encoding. This encoding is also referred to as latin1.

you can find it in "ouput" section there: http://dev.maxmind.com/geoip/web-services

looking for a sample response...

Olivier Gonzalez

Here is a sample result string obtained with raw_data.inspect :

"FR,A3,Ch\xE2lo-saint-mars,,48.427601,2.065400,0,0,\"Free SAS\",\"Free SAS\""

As you may see, maxmind documentation provide a ruby 1.9 example and they use:

response.body.encode('utf-8', 'iso-8859-1').parse_csv

.encode('utf-8', 'iso-8859-1').parse_csv works well when applied to raw_data string, so you may use it I think

I don't know about ruby 1.8 :/

Alex Reisner
Owner

Thanks for the sample data! Unfortunately neither the current solution nor theirs seems to work!

I wonder if Geocoder should be doing any conversion at all. Granted, among us westerners UTF-8 seems to be the current encoding of choice, but some people might want to convert to something else (eg: UTF-16 or -32), in which case this would be some wasted CPU cycles.

I do think it's worthwhile, though. Most of the other supported services (including Google and Bing) return UTF-8 and perhaps more care should be given to ensure that all results are UTF-8 encoded.

However, I am not going to do the conversion with Ruby 1.8.x. Unless I'm mistaken, every method of changing character encoding in Ruby 1.8.x requires some external system dependency (like iconv), which is too much complexity for Geocoder. I don't like this inconsistency between Ruby versions, and will make a note of it, but I'm not going to introduce a large dependency for what is sort of a minor issue.

Alex Reisner
Owner

Also, what's the reasoning behind the substrings in Geocoder::Result::Maxmind#isp and #organization? Those values appear to be getting truncated.

Olivier Gonzalez

hmm, I didn't catch that. To be honest I don't remember why I did such a thing (I should have written a comment), but I think it was mandatory to get clean values ^^

isp and organization are the only attribute that are wrapped with double quotes in the result string. I suppose there was something weird with this and I was forced to manually trim them.

It seems that it's not the case anymore so it should be dropped as they are indeed cutting values now.

Olivier Gonzalez

Ah, one more thing about my code: I only focused on supporting the "City/ISP/Org" service (the one we were using).

But Maxmind also provides other services which contain less or more attributes, making the current code failing for them.

This is due to the fact that Maxmind response length is different for each services, see http://dev.maxmind.com/geoip/web-services#Output_field_order-3

So it's worth warning user about this or implementing other services for full support.

thanks

Alex Reisner
Owner

Thanks for pointing out the different response types. Sounds like we need to support all four types. Here's what I'm going to do:

  1. Fix the isp/organization trimming.
  2. Implement configuration/support for the four MaxMind services/response types.
  3. Remove string encoding conversion.

The string encoding is too complicated to implement correctly/consistently before the 1.1.6 release (which I'd like to do today or tomorrow). I'll create an issue about handling encoding from all APIs consistently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 12, 2011
  1. Olivier Gonzalez
Commits on Oct 13, 2011
  1. Olivier Gonzalez
Commits on Apr 30, 2012
  1. Olivier Gonzalez
  2. Olivier Gonzalez
Commits on May 11, 2012
  1. Olivier Gonzalez
This page is out of date. Refresh to see the latest.
16 README.rdoc
View
@@ -235,6 +235,12 @@ By default Geocoder uses Google's geocoding API to fetch coordinates and street
# to use an API key:
Geocoder::Configuration.api_key = "..."
+ # IP geocoding service :
+ Geocoder::Configuration.ip_lookup = :maxmind
+
+ # to use an API key for IP geocoding service:
+ Geocoder::Configuration.ip_lookup_api_key = "..."
+
# geocoding service request timeout, in seconds (default 3):
Geocoder::Configuration.timeout = 5
@@ -339,6 +345,16 @@ Documentation:: http://github.com/fiorix/freegeoip/blob/master/README.rst
Terms of Service:: ?
Limitations:: ?
+==== MaxMind City
+
+API key:: required
+Quota:: Requests Packs can be purchased (starting at 50,000 for 20$)
+Region:: world
+SSL support:: no
+Languages:: English
+Documentation:: http://www.maxmind.com/app/web_services
+Terms of Service:: ?
+Limitations:: ?
== Caching
4 lib/geocoder.rb
View
@@ -64,7 +64,7 @@ def street_lookups
# All IP address lookups, default first.
#
def ip_lookups
- [:freegeoip]
+ [:freegeoip, :maxmind]
end
@@ -77,7 +77,7 @@ def ip_lookups
#
def lookup(query)
if ip_address?(query)
- get_lookup(ip_lookups.first)
+ get_lookup(Configuration.ip_lookup || ip_lookups.first)
else
get_lookup(Configuration.lookup || street_lookups.first)
end
8 lib/geocoder/configuration.rb
View
@@ -28,6 +28,14 @@ def self.options_and_defaults
# for Google Premier use a 3-element array: [key, client, channel]
[:api_key, nil],
+ # name of IP geocoding service (symbol)
+ # FIXME: temporary added for Maxmind support, need clean rewrite
+ [:ip_lookup, nil],
+
+ # API key for IP geocoding service
+ # FIXME: temporary added for Maxmind support, need clean rewrite
+ [:ip_lookup_api_key, nil],
+
# cache object (must respond to #[], #[]=, and #keys)
[:cache, nil],
41 lib/geocoder/lookups/maxmind.rb
View
@@ -0,0 +1,41 @@
+require 'geocoder/lookups/base'
+require 'geocoder/results/maxmind'
+require 'csv'
+
+module Geocoder::Lookup
+ class Maxmind < Base
+
+ private # ---------------------------------------------------------------
+
+ def results(query, reverse = false)
+ # don't look up a loopback address, just return the stored result
+ return [reserved_result] if loopback_address?(query)
+ begin
+ doc = fetch_data(query, reverse)
+ if doc && doc.size == 10
+ return [doc]
+ else
+ warn "Maxmind error : #{doc[10]}" if doc
+ return []
+ end
+ rescue StandardError => err
+ raise_error(err)
+ return []
+ end
+ end
+
+ def parse_raw_data(raw_data)
+ # Maxmind just returns text/plain as csv format but according to documentation,
+ # we get ISO-8859-1 encoded string. We need to convert it.
+ CSV.parse_line raw_data.force_encoding("ISO-8859-1").encode("UTF-8")
+ end
+
+ def reserved_result
+ ",,,,0,0,0,0,,"
+ end
+
+ def query_url(query, reverse = false)
+ "http://geoip3.maxmind.com/f?l=#{Geocoder::Configuration.ip_lookup_api_key}&i=#{query}"
+ end
+ end
+end
55 lib/geocoder/results/maxmind.rb
View
@@ -0,0 +1,55 @@
+require 'geocoder/results/base'
+
+module Geocoder::Result
+ class Maxmind < Base
+
+ def address(format = :full)
+ s = state_code.to_s == "" ? "" : ", #{state_code}"
+ "#{city}#{s} #{postal_code}, #{country_code}".sub(/^[ ,]*/, "")
+ end
+
+ def country_code
+ @data[0]
+ end
+
+ def state_code
+ @data[1]
+ end
+
+ def city
+ @data[2]
+ end
+
+ def postal_code
+ @data[3]
+ end
+
+ def coordinates
+ [@data[4].to_f, @data[5].to_f]
+ end
+
+ def metrocode
+ @data[6]
+ end
+
+ def area_code
+ @data[7]
+ end
+
+ def isp
+ @data[8][1,@data[8].length-2]
+ end
+
+ def organization
+ @data[9][1,@data[9].length-2]
+ end
+
+ def country #not given by MaxMind
+ country_code
+ end
+
+ def state #not given by MaxMind
+ state_code
+ end
+ end
+end
1  test/fixtures/maxmind_74_200_247_59.txt
View
@@ -0,0 +1 @@
+US,TX,Plano,75093,33.034698,-96.813400,623,972,"Layered Technologies , US","Layered Technologies , US"
1  test/fixtures/maxmind_no_results.txt
View
@@ -0,0 +1 @@
+,,,,,,,,,,IP_NOT_FOUND
6 test/lookup_test.rb
View
@@ -27,4 +27,10 @@ def test_yahoo_app_id
g = Geocoder::Lookup::Yahoo.new
assert_match "appid=MY_KEY", g.send(:query_url, "Madison Square Garden, New York, NY 10001, United States")
end
+
+ def test_maxmind_api_key
+ Geocoder::Configuration.ip_lookup_api_key = "MY_KEY"
+ g = Geocoder::Lookup::Maxmind.new
+ assert_match "l=MY_KEY", g.send(:query_url, "74.200.247.59")
+ end
end
14 test/services_test.rb
View
@@ -94,6 +94,20 @@ def test_freegeoip_result_components
assert_equal "Plano, TX 75093, United States", result.address
end
+ # --- MaxMind ---
+
+ def test_maxmind_result_on_ip_address_search
+ Geocoder::Configuration.ip_lookup = :maxmind
+ result = Geocoder.search("74.200.247.59").first
+ assert result.is_a?(Geocoder::Result::Maxmind)
+ end
+
+ def test_maxmind_result_components
+ Geocoder::Configuration.ip_lookup = :maxmind
+ result = Geocoder.search("74.200.247.59").first
+ assert_equal "Plano, TX 75093, US", result.address
+ end
+
# --- Bing ---
13 test/test_helper.rb
View
@@ -133,6 +133,19 @@ def fetch_raw_data(query, reverse = false)
end
end
+ class Maxmind < Base
+ private #-----------------------------------------------------------------
+ def fetch_raw_data(query, reverse = false)
+ raise TimeoutError if query == "timeout"
+ raise SocketError if query == "socket_error"
+ file = case query
+ when "no results"; :no_results
+ else "74_200_247_59"
+ end
+ read_fixture "maxmind_#{file}.txt"
+ end
+ end
+
class Bing < Base
private #-----------------------------------------------------------------
def fetch_raw_data(query, reverse = false)
Something went wrong with that request. Please try again.