Restructure Zippopotamus Postal Code

Moved to MONGODB

ADDED
-----

[scripts/add.py] Script for adding csv files to mongodb
[scrips/remove.py] Script for removing csv files from db
[scripts/export.bash] Exports data country by country from db
[scripts/table.py] builds html table from db information

[appendix/country.txt] This is a list of all countries supported
[appendix/genonames-headers.txt] A list of headers for geonames data
[appendx/headers.txt] A list of Zippopotamus Headers

DELETED
-------

raw/* : All dump files have been moved.

The intention of the raw folder, is for crowdsourced data to be collected

appendix/country.txt

@@ -0,0 +1,70 @@
+AD,Andorra
+AR,Argentina
+AS,American Samoa
+AT,Austria
+AU,Australia
+BD,Bangladesh
+BE,Belgium
+BG,Bulgaria
+BR,Brazil
+CA,Candada
+CH,Switzerland
+CZ,Czech Republic
+DE,Germany
+DK,Denmark
+DO,Dominican Republic
+ES,Spain
+FI,Finland
+FO,Faroe Islands
+FR,France
+GB,Great Britain
+GF,French Guyana
+GG,Guernsey
+GL,Greenland
+GP,Guadeloupe
+GT,Guatemala
+GU,Guam
+GY,Guyana
+HR,Croatia
+HU,Hungary
+IM,Isle of Man
+IN,India
+IS,Iceland
+IT,Italy
+JE,Jersey
+JP,Japan
+LI,Liechtenstein
+LK,Sri Lanka
+LT,Lithuania
+LU,Luxembourg
+MC,Monaco
+MD,Moldavia
+MH,Marshall Islands
+MK,Macedonia
+MP,Northern Mariana Islands
+MQ,Martinique
+MX,Mexico
+MY,Malaysia
+NL,Holland
+NO,Norway
+NZ,New Zealand
+PH,Phillippines
+PK,Pakistan
+PL,Poland
+PM,Saint Pierre and Miquelon
+PR,Puerto Rico
+PT,Portugal
+RE,French Reunion
+RU,Russia
+SE,Sweden
+SI,Slovenia
+SJ,Svalbard & Jan Mayen Islands
+SK,Slovak Republic
+SM,San Marino
+TH,Thailand
+TR,Turkey
+US,United States
+VA,Vatican
+VI,Virgin Islands
+YT,Mayotte
+ZA,South Africa

appendix/geonames-headers.txt

@@ -0,0 +1 @@
+country abbreviation,post code,place name,state,state abbreviation,ignore1,ignore2,ignore3,ignore4,latitude,longitude,ignore5

appendix/headers.txt

@@ -0,0 +1 @@
+country,country abbreviation,post code,place name,state,state abbreviation,latitude,longitude

scripts/add.py

@@ -0,0 +1,59 @@
+import sys
+import os
+import csv
+
+from pymongo import Connection
+from pymongo.database import Database
+
+appendix_file = '../appendix/country.txt'
+header_file = '../appendix/headers.txt'
+
+
+def add(csv_file):
+
+    # Generate N Records
+    with open( csv_file , 'rb')  as f:
+        reader = csv.reader( f , delimiter=',', quotechar='"') 
+        headers = reader.next()
+        for row in reader:
+            record = dict()
+
+            for ii in range(0, len(headers)):
+                if 'ignore' not in headers[ii]:
+                    record[ headers[ii] ] = unicode( row[ii], 'utf-8' )
+            save(record)
+
+    pass
+
+def save( record ):
+    db['global'].save(record)
+
+if __name__ == "__main__" :
+
+    if ( len(sys.argv) != 5 ):
+        print "Usage: %s <username> <password> <url> <csv-files>" % sys.argv[0]
+
+    username = sys.argv[1]
+    password = sys.argv[2]
+    url  =     sys.argv[3]
+    file_list = sys.argv[4:]
+
+    print username
+    print password
+    print len(file_list)
+
+    if ( len(url) == 0 ):
+        connection = Connection()              # Connect to localhost
+    else:
+        connection = Connection( url )         # Connect to remote db
+
+    db = Database(connection,'zip')              # Get zip database
+    db.authenticate(username,password)           # Authenticate
+
+
+    for csv_file in file_list:  # Add all the files 
+        add( csv_file )
+
+
+
+

scripts/export.bash

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# Check that we have at least n arguments
+
+if [  $# -ne 4 ] 
+then
+	echo "Please enter <append> <raw-dump-dir> <user> <pass>"
+	exit -1
+fi
+
+
+# Get directory and name
+appendix=$1
+raw=$2
+user=$3
+pass=$4
+
+echo "Dumping $appendix into $raw.txt"
+
+while read line
+do
+	echo -e "$line \n"
+	name=${line:0:2}
+	mongoexport --db zip  --collection global -u $user -p $pass -q "{'country abbreviation':'$name'}" -f 'country','country abbreviation','post code','place name','state','state abbreviation','latitude','longitude' --csv -o $raw$name.txt
+done < $appendix
+