Skip to content
This repository

Modular Street Address Geocoder

branch: master

Merge pull request #31 from theduckylittle/fixing_flipped

Fixing the "flipped" status of a record, Removed "excensus" branch notice -
thanks for the edits!
latest commit 2500432a06
Andrew Turner ajturner authored
Octocat-spinner-32 bin Parameterized rebuild_metaphones December 23, 2011
Octocat-spinner-32 build Probably too many changes for one commit but here they are: December 23, 2011
Octocat-spinner-32 conf Fix upstart job. June 08, 2011
Octocat-spinner-32 debian Prepared changelog for build 66 June 18, 2011
Octocat-spinner-32 demos Fix a bug in edge interpolation caused by zero-length segments. June 16, 2011
Octocat-spinner-32 doc (and the Makefile too) March 19, 2009
Octocat-spinner-32 lib Found some issues with the flipped logic thanks to the type-lessness … August 13, 2012
Octocat-spinner-32 navteq Add README for Navteq import and fix PATH. June 11, 2009
Octocat-spinner-32 src Added an install parameter to the Makefile December 27, 2011
Octocat-spinner-32 test Fixed problem with removing city name from street when already parsed… May 17, 2012
Octocat-spinner-32 .gitignore ignore import log files October 14, 2011
Octocat-spinner-32 History.txt Add a whole bunch of stuff towards Ruby packaging. June 05, 2009
Octocat-spinner-32 LICENSE.txt Merging in changes from Pete Warden on updating CSV libraries and com… March 23, 2011
Octocat-spinner-32 Makefile Remove rubygems references and fix a bunch of other stuff. August 23, 2010
Octocat-spinner-32 Manifest.txt switched references from README.txt to README.rdoc October 19, 2009
Octocat-spinner-32 README.rdoc Removed my notes about Excensus branch from the docs as the branch ha… August 13, 2012
Octocat-spinner-32 REST.rdoc Finish tweaking the GeoJSON output and update the docs. August 23, 2010
Octocat-spinner-32 TODO.txt Substantially update README.txt. June 05, 2009
Octocat-spinner-32 gemspec Update gemspec to allow installation on ruby 1.8.6 systems June 18, 2012
Octocat-spinner-32 setup.rb Add setup.rb. August 07, 2010
README.rdoc

Geocoder::US

Geocoder::US 2.0 is a software package designed to geocode US street addresses. Although it is primarily intended for use with the US Census Bureau's free TIGER/Line dataset, it uses an abstract US address data model that can be employed with other sources of US street address range data.

Geocoder::US 2.0 implements a Ruby interface to parse US street addresses, and perform fuzzy lookup against an SQLite 3 database. Geocoder::US is designed to return the best matches found, with geographic coordinates interpolated from the street range dataset. Geocoder::US will fill in missing information, and it knows about standard and common non-standard postal abbreviations, ordinal versus cardinal numbers, and more.

Geocoder::US 2.0 is shipped with a free US ZIP code data set, compiled from public domain sources.

Synopsis

>> require 'geocoder/us'
>> db = Geocoder::US::Database.new("/opt/tiger/geocoder.db")
>> p db.geocode("1600 Pennsylvania Av, Washington DC")

[{:pretyp=>"", :street=>"Pennsylvania", :sufdir=>"NW", :zip=>"20502",
  :lon=>-77.037528, :number=>"1600", :fips_county=>"11001", :predir=>"",
  :precision=>:range, :city=>"Washington", :lat=>38.898746, :suftyp=>"Ave",
  :state=>"DC", :prequal=>"", :sufqual=>"", :score=>0.906, :prenum=>""}]

Web service

There is a small example webservice included in the gem at lib/geocoder/rest.rb. It requires Sinatra to run as well as setting the GEOCODER_DB environment variable. To run it:

GEOCODER_DB=/path/to/geocoder/geocoder.db ruby /path/to/gem/geocoder/lib/geocoder/us/rest.rb

You can then query the GeoCommons geocoder by calling the web service like:

http://localhost:9393/geocode.html?q=303+11th St NE,Washington,DC

Optional Options

dbtype Geocoder::US::Database.new(“/opt/tiger/geocoder.db”, {:dbtype => 1})

The dbtype option is used when your datasbase encodes it's geometry blogs according to a special format. The first and default value, is in a series of little-endian 4-byte ints (“V*”) The second is for ('CVVD*') format. Use option value 2 for this second type

debug Geocoder::US::Database.new(“/opt/tiger/geocoder.db”, {:debug => false})

cache Geocoder::US::Database.new(“/opt/tiger/geocoder.db”, {:cache => 50000}) The cache_size argument is measured in kilobytes and is used to set the SQLite cache size;

larger values will trade memory for speed in long-running processes.

Prerequisites

To build Geocoder::US, you will need gcc/g++, make, bash or equivalent, the standard *NIX 'unzip' utility, and the SQLite 3 executable and development files installed on your system.

To use the Ruby interface, you will need the 'Text' gem installed from rubyforge. To run the tests, you will also need the 'fastercsv' gem.

Additionally, you will need a custom build of the 'sqlite3-ruby' gem that supports loading extension modules in SQLite. You can get a patched version of this gem from github.com/schuyler/sqlite3-ruby/. Until the sqlite3-ruby maintainers roll in the relevant patch, you will need this version.

NOTE: If you do not have /usr/include/sqlite3ext.h installed, then your sqlite3 binaries are probably not configured to support dynamic extension loading. If not, you must compile and install SQLite from source, or rebuild your system packages. This is not believed to be a problem on Debian/Ubuntu, but is known to be a problem with Red Hat/CentOS.

NOTE: If you do have to install from source, make sure that the source-installed 'sqlite3' program is in your path before proceeding (and not the system-installed version), using `which sqlite3`. Also, be sure that you've added your source install prefix (usually /usr/local) to /etc/ld.so.conf (or its moral equivalent) and that you've run /sbin/ldconfig.

Thread safety

SQLite 3 is not designed for concurrent use of a single database handle across multiple threads. Therefore, to prevent segfaults, Geocoder::US::Database implements a global mutex that wraps all database access. The use of this mutex will ensure stability in multi-threaded applications, but incurs a performance penalty. However, since the database is read-only from Ruby, there's no reason in principle why multi-threaded apps can't each have their own database handle.

To disable the mutex for better performance, you can do the following:

* Read the following and make sure you understand them:
   * http://www.sqlite.org/faq.html#q6
   * http://www.sqlite.org/cvstrac/wiki?p=MultiThreading
* Make sure you have compiled SQLite 3 with thread safety enabled.
* Instantiate a separate Geocoder::US::Database object for *each* thread
  in your Ruby script, and pass :threadsafe => true to new() to disable mutex
  synchronization.

Per the SQLite 3 documentation, do not attempt to retain a Geocoder::US::Database object across a fork! “Problems will result if you do.”

Building Geocoder::US

Unpack the source and run 'make'. This will compile the SQLite 3 extension needed by Geocoder::US, the Shapefile import utility, and the Geocoder-US gem.

You can run 'make install' as root to install the gem systemwide.

Generating a Geocoder::US Database

Build the package from source as described above. Generating the database involves three basic steps:

  • Import the Shapefile data into an SQLite database.

  • Build the database indexes.

  • Optionally, rebuild the database to cluster indexed rows.

We will presume that you are building a Geocoder::US database from TIGER/Line, and that you have obtained the complete set of TIGER/Line ZIP files, and put the entire tree in /opt/tiger. Please adjust these instructions as needed.

A full TIGER/Line database import takes ten hours to run on a normal Amazon EC2 instance, and takes up a little over 5 gigabytes after all is said and done. You will need to have at least 12 gigabytes of free disk space after downloading the TIGER/Line dataset, if you are building the full database.

Import TIGER/Line

From inside the Geocoder::US source tree, run the following:

$ build/tiger_import /opt/tiger/geocoder.db /opt/tiger

This will unpack each TIGER/Line ZIP file to a temporary directory, and perform the extract/transform/load sequence to incrementally build the database. The process takes about 10-12 hours on a normal Amazon EC2 instance, or about 5 CPU hours flat out on a modern PC. Note that not all TIGER/Line source files contain address range information, so you will see error messages for some counties, but this is normal.

If you only want to import specific counties, you can pipe a list of TIGER/Line county directories to tiger_import on stdin. For example, the following will install just the data for the state of Delaware:

$ ls -d /opt/tiger/10_DELAWARE/1* | build/tiger_import ~/delaware.db

The tiger_import process uses a binary utility, shp2sqlite, which is derived from shp2pgsql, which ships with PostGIS. The shp2sqlite utility converts .shp and .dbf files into SQL suitable for import into SQLite. This SQL is then piped into the sqlite3 command line tool, where it is loaded into temporary tables, and then a set of static SQL statements (kept in the sql/ directory) are used to transform this data and import it into the database itself.

Build metaphones using Ruby metaphone

run bin/rebuild_metaphones /opt/tiger/geocoder.db

This creates the metaphones using Ruby's metaphone function and will produce better geocoding results.

Build the indexes

After the database import is complete, you will want to construct the database indexes:

$ build/build_indexes /opt/tiger/geocoder.db

This process takes 25 minutes on an EC2 instance (8 CPU minutes), but it's a lot faster than building the indexes incrementally during the import process. Basically, this process simply feeds SQL statements to the sqlite3 utility to construct the indexes on the existing database.

Cluster the database tables (optional)

As a final optional step, you can cluster the database tables according to their indexes, which will make the database smaller, and lookups faster. This process will take an hour or two, and may be a micro-optimization.

$ build/rebuild_cluster /opt/tiger/geocoder.db

You will need as much free disk space to run rebuild_cluster as the database takes up, because the process essentially reconstructs the database in a new file, and then it renames the new database over top of the old.

Running the unit tests

From within the source tree, you can run the following:

$ ruby tests/run.rb

This tests the libraries, except for the database routines. If you have a database built, you can run the test harness like so:

$ ruby tests/run.rb /opt/tiger/geocoder.db

The full test suite may take 30 or so seconds to run completely.

License

Geocoder::US 2.0 was based on earlier work by Schuyler Erle on a Perl module of the same name. You can find it at search.cpan.org/~sderle/.

Geocoder::US 2.0 was written by Schuyler Erle, of Entropy Free LLC, with the gracious support of FortiusOne, Inc. Please send bug reports, patches, kudos, etc. to patches at geocoder.us.

Copyright © 2009 FortiusOne, Inc.

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.
Something went wrong with that request. Please try again.