Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

- Data pulls from GeoNames and imports directly into database, no fix…

…tures

- Downloads and updates only if data has changed
- All ids are same as GeoNames', so updates only overwrite matching ids
- Localized / Alternate names and dynamically created locale models
- Main geo name is now ASCII instead of UTF-8 (better for coding / search)
- The UTF-8 standard name is still available as a fallback if name isn't localized
- Using cities5000 instead of cities1000 (half the size)
- Fast imports using dict index instead of db
- Fixed latitude / longitude mixup
- Full admin interface
- Reduced to one command 'cities' with options
- Cleaned up code and logging
  • Loading branch information...
commit 48d2c2e37c0febc436414b441a95925fedd5fbb8 1 parent 991f581
@Qarterd Qarterd authored
View
6 .gitignore
@@ -1 +1,5 @@
-*.pyc
+*.pyc
+
+*.egg-info
+build
+dist
View
0  MANIFEST.in
No changes.
View
69 README
@@ -1,25 +1,24 @@
django-cities - Place models and data for Django apps
=====================================================
-This project includes country, region, city, and district models and is
-prepopulated with data from GeoNames. The GeoNames import script
-is also included if you'd like to re-import the data yourself, or change
-the way in which it gets cleaned up.
+This add-on provides models and commands to import country / region / city / district data into your database.
+The data is pulled from GeoNames and contains localized names, codes, geo-coords, and population.
-Includes 234 counties, 2,610 regions, 97,949 cities and 606 districts
+Your database must support spatial queries, see GeoDjango for setup.
-For more information and examples see http://www.coderholic.com/django-cities-countries-regions-cities-and-districts-for-django/
+For more information see: http://www.coderholic.com/django-cities-countries-regions-cities-and-districts-for-django/
-Examples:
+
+Examples
=========
Finding all London boroughs:
>>> london = City.objects.filter(country__name='United Kingdom').get(name='London')
>>> boroughs = District.objects.filter(city=london)
-Nearest city to a given lat,lon:
+Nearest city to a given geo-coord (longitude, latitude):
->>> City.objects.nearest_to(51, 1)
+>>> City.objects.distance(Point(1, 51)).order_by('distance')[0]
<City: Dymchurch, Kent, United Kingdom>
5 Nearest cities to London:
@@ -27,27 +26,55 @@ Nearest city to a given lat,lon:
>>> london = City.objects.filter(country__name='United Kingdom').get(name='London')
>>> nearest = City.objects.distance(london.location).exclude(id=london.id).order_by('distance')[:5]
-Important Notice:
+Get all countries in Japanese preferring official names if available, fallback on ASCII names:
+
+>>> [country.alt_names_ja.get_preferred(default=country.name) for country in Country.objects.all()]
-The old dump file contained wrong locations (longtitude and latitude upside down, to be precise). The latest dump file fixed this error. Before updating to the latest dump, please read the following changes made to the current release:
+Use alternate names model to get Vancouver in Japanese:
-1. The dump file is renamed from initial_data.json to geonames_dump.json. This means data won't be loaded into databases automatically when running syncdb command. To load the dump, you have to run "manage.py loaddata geonames_dump" explicitly, which is actually a good feature since the original name can cause this dump file to be loaded every time you run syncdb command. There are not just only a few records in it, you know...
+>>> geo_alt_names[City]['ja'].objects.get_preferred(geo__name='Vancouver', default='Vancouver')
-2. The dump file has the correct locations now. Distances will be calculated correctly.
+Gather names of Tokyo from all LANGUAGES, including names from undetermined language data ('und'):
-3. A command called load_geonames is added. This command allows you to only load countries you feel needed in your application. Originally, the whole dump is loaded into the database, which might be too overwhelming for you or your database. Now, say you only want data for United States and Canada, run "manage.py load_geonames US CA", dada... only districts, cities, regions and countries of these two countries are loaded into the database. Later you decide to add China to the country list, you run "manage.py load_geonames CN". Now you have data for three countries.
+>>> [name for locale in settings.CITIES_LOCALES
+ for name in geo_alt_names[City][locale].objects.filter(geo__name='Tokyo')]
-In short, after installing django-cities, add django-cities to your installed app in settings.py. Then you have two choices:
-1. To import the whole dump, run command:
+Install
+=========
+- Run: python setup.py install
+- Add/Merge the following into your settings.py:
-manage.py loaddata geonames_dump
+INSTALLED_APPS = (
+ 'cities',
+)
-2. Choose certain countries to import, run command:
+LOGGING = {
+ 'handlers': {
+ 'console':{
+ 'level':'DEBUG',
+ 'class':'logging.StreamHandler',
+ },
+ },
+ 'loggers': {
+ 'cities': {
+ 'handlers': ['console'],
+ 'level': 'INFO',
+ }
+ }
+}
-manage.py load_geonames [country_code_list]
+CITIES_LOCALES = ['und']
+if 'LANGUAGES' in locals(): CITIES_LOCALES += [lang[0] for lang in LANGUAGES]
-country_code_list is a list of country codes separated by spaces.
-If you've installed django-cities and imported data before, the first command will fix all the location errors and the second one will fix the location errors of selected countries.
+- Sync your database with the new models: 'manage.py syncdb'
+- Populate or update your database by running: 'manage.py cities'
+
+Notes
+=========
+The localized models / tables are created dynamically from CITIES_LOCALES.
+Some datasets are very large (> 100 MB) and take time to download / import, and there's no progress display.
+Data will only be downloaded / imported if it is newer than your data, and only matching rows will be overwritten.
+The cities manage command has options, see --help. Verbosity is controlled through LOGGING.
View
43 cities/admin.py
@@ -1,10 +1,39 @@
from django.contrib import admin
-from models import * #@UnusedWildImport
+from models import *
-class SearchableAdmin(admin.ModelAdmin):
- search_fields = ['name']
+class CountryAdmin(admin.ModelAdmin):
+ list_display = ['name', 'code', 'tld', 'population']
+ search_fields = ['name']
+
+admin.site.register(Country, CountryAdmin)
-admin.site.register(Country, SearchableAdmin)
-admin.site.register(Region, SearchableAdmin)
-admin.site.register(City, SearchableAdmin)
-admin.site.register(District, SearchableAdmin)
+class RegionAdmin(admin.ModelAdmin):
+ ordering = ['name_std']
+ list_display = ['name_std', 'country', 'code']
+ search_fields = ['name', 'name_std']
+
+admin.site.register(Region, RegionAdmin)
+
+class CityAdmin(admin.ModelAdmin):
+ ordering = ['name_std']
+ list_display = ['name_std', 'region', 'population']
+ search_fields = ['name', 'name_std']
+
+admin.site.register(City, CityAdmin)
+
+class DistrictAdmin(admin.ModelAdmin):
+ ordering = ['name_std']
+ list_display = ['name_std', 'city', 'population']
+ search_fields = ['name', 'name_std']
+ readonly_fields = ['city'] # City choice list creation is slow
+
+admin.site.register(District, DistrictAdmin)
+
+class GeoAltNameAdmin(admin.ModelAdmin):
+ ordering = ['name']
+ list_display = ['name', 'geo', 'is_preferred', 'is_short']
+ list_filter = ['is_preferred', 'is_short']
+ search_fields = ['name']
+ readonly_fields = ['geo'] # City choice list creation is slow
+
+[admin.site.register(geo_alt_name, GeoAltNameAdmin) for locales in geo_alt_names.values() for geo_alt_name in locales.values()]
View
1,112,781 cities/fixtures/geonames_dump.json
0 additions, 1,112,781 deletions not shown
View
259 cities/management/commands/cities.py
@@ -0,0 +1,259 @@
+"""
+GeoNames city data import script.
+Requires the following files:
+
+http://download.geonames.org/export/dump/
+- Countries: countryInfo.txt
+- Regions: admin1CodesASCII.txt
+- Cities / Districts: cities5000.zip
+- Localization: alternateNames.zip
+"""
+
+import os
+import sys
+import urllib
+import logging
+import zipfile
+import time
+from optparse import make_option
+from django.core.management.base import BaseCommand
+from django.template.defaultfilters import slugify
+from ...models import *
+
+class Command(BaseCommand):
+ app_dir = os.path.normpath(os.path.dirname(os.path.realpath(__file__)) + '/../..')
+ data_dir = os.path.join(app_dir, 'data')
+ logger = logging.getLogger("cities")
+ url_base = 'http://download.geonames.org/export/dump/'
+
+ option_list = BaseCommand.option_list + (
+ make_option('--force', action='store_true', default=False,
+ help='Import even if files are up-to-date.'),
+ )
+
+ def handle(self, *args, **options):
+ self.options = options
+ self.import_countries()
+ self.import_regions()
+ self.import_cities()
+ self.import_geo_alt_name()
+
+ def download(self, filename):
+ web_file = urllib.urlopen(self.url_base + filename)
+ web_file_time = time.strptime(web_file.headers['last-modified'], '%a, %d %b %Y %H:%M:%S %Z')
+ web_file_size = int(web_file.headers['content-length'])
+
+ file_exists = False
+ try:
+ file_time = time.gmtime(os.path.getmtime(os.path.join(self.data_dir, filename)))
+ file_size = os.path.getsize(os.path.join(self.data_dir, filename))
+ if file_time >= web_file_time and file_size == web_file_size:
+ self.logger.info("File up-to-date: " + filename)
+ file_exists = True
+ if not self.options['force']: return None
+ except: pass
+
+ if not file_exists:
+ self.logger.info("Downloading: " + filename)
+ if not os.path.exists(self.data_dir):
+ os.makedirs(self.data_dir)
+ file = open(os.path.join(self.data_dir, filename), 'w+b')
+ file.write(web_file.read())
+ file.seek(0)
+ else:
+ file = open(os.path.join(self.data_dir, filename), 'rb')
+
+ return file
+
+ def import_countries(self):
+ file = self.download('countryInfo.txt')
+ if not file: return
+
+ self.logger.info("Importing country data")
+ for line in file:
+ if len(line) < 1 or line[0] == '#': continue
+ items = [e.strip() for e in line.decode("utf-8").split('\t')]
+
+ country = Country()
+ try: country.id = int(items[16])
+ except: continue
+ country.name = items[4]
+ country.slug = slugify(country.name)
+ country.code = items[0]
+ country.population = items[7]
+ country.continent = items[8]
+ country.tld = items[9][1:] # strip the leading .
+
+ country.save()
+ self.logger.debug(u"Added country: {}, {}".format(country.code, country))
+
+ file.close()
+
+ def import_regions(self):
+ file = self.download('admin1CodesASCII.txt')
+ if not file: return
+
+ self.logger.info("Building country index")
+ country_objects = {}
+ for obj in Country.objects.all():
+ country_objects[obj.code] = obj
+
+ self.logger.info("Importing region data")
+ for line in file:
+ if len(line) < 1 or line[0] == '#': continue
+ items = [e.strip() for e in line.split('\t')]
+
+ region = Region()
+ region.id = items[3]
+ region.name = items[2]
+ region.name_std = items[1]
+ region.slug = slugify(region.name)
+ region.code = items[0]
+ try: region.country = country_objects[region.code[:2]]
+ except:
+ self.logger.warning(u"Region: {}: Cannot find country: {} -- skipping".format(region.name, region.code[:2]))
+ continue
+
+ region.save()
+ self.logger.debug(u"Added region: {}, {}".format(region.code, region))
+
+ file.close()
+
+ def import_cities(self):
+ file = self.download('cities5000.zip')
+ if not file: return
+
+ zip = zipfile.ZipFile(file)
+ data = zip.read('cities5000.txt').split('\n')
+ zip.close()
+ file.close()
+
+ self.logger.info("Building region index")
+ region_objects = {}
+ for obj in Region.objects.all():
+ region_objects[obj.code] = obj
+
+ self.logger.info("Importing city data")
+ for line in data:
+ if len(line) < 1 or line[0] == '#': continue
+ items = [e.strip() for e in line.split('\t')]
+
+ admin_type = items[11]
+ type = items[7]
+
+ # See http://www.geonames.org/export/codes.html
+ if type not in ['PPL', 'PPLA', 'PPLC', 'PPLA2', 'PPLA3', 'PPLA4']: continue
+
+ city = City()
+ city.id = items[0]
+ city.name = items[2]
+ city.name_std = items[1]
+ city.slug = slugify(city.name)
+ city.location = Point(float(items[5]), float(items[4]))
+ city.population = items[14]
+
+ # Try more specific region first
+ region = None
+ if items[11]:
+ try:
+ code = "{}.{}".format(items[8], items[11])
+ region = region_objects[code]
+ except: pass
+ if not region:
+ try:
+ code = "{}.{}".format(items[8], items[10])
+ region = region_objects[code]
+ except:
+ self.logger.warning(u"City: {}: Cannot find region: {} -- skipping".format(city.name, code))
+ continue
+
+ city.region = region
+ city.save()
+ self.logger.debug(u"Added city: {}".format(city))
+
+ self.import_districts(data, region_objects)
+
+ def import_districts(self, data, region_objects):
+
+ self.logger.info("Importing district data")
+ for line in data:
+ if len(line) < 1 or line[0] == '#': continue
+ items = [e.strip() for e in line.split('\t')]
+
+ admin_type = items[11]
+ type = items[7]
+
+ # See http://www.geonames.org/export/codes.html
+ if type not in ['PPLX']: continue
+
+ district = District()
+ district.id = items[0]
+ district.name = items[2]
+ district.name_std = items[1]
+ district.slug = slugify(district.name)
+ district.location = Point(float(items[5]), float(items[4]))
+ district.population = items[14]
+
+ # Try more specific region first
+ region = None
+ if items[11]:
+ try:
+ code = "{}.{}".format(items[8], items[11])
+ region = region_objects[code]
+ except: pass
+ if not region:
+ try:
+ code = "{}.{}".format(items[8], items[10])
+ region = region_objects[code]
+ except:
+ self.logger.warning(u"District: {}: Cannot find region: {} -- skipping".format(district.name, code))
+ continue
+
+ # Set the nearest city
+ district.city = City.objects.filter(population__gt=125000).distance(district.location).order_by('distance')[0]
+ district.save()
+ self.logger.debug(u"Added district: {}".format(district))
+
+ def import_geo_alt_name(self):
+ file = self.download('alternateNames.zip')
+ if not file: return
+
+ zip = zipfile.ZipFile(file)
+ data = zip.read('alternateNames.txt').split('\n')
+ zip.close()
+ file.close()
+
+ self.logger.info("Building geo index")
+ geo_id_info = {}
+ for type_ in geo_alt_names:
+ for obj in type_.objects.all():
+ geo_id_info[obj.id] = {
+ 'type': type_,
+ 'object': obj,
+ }
+
+ self.logger.info("Importing alternate name data")
+ for line in data:
+ if len(line) < 1 or line[0] == '#': continue
+ items = [e.strip() for e in line.split('\t')]
+
+ # Only get names for languages in use
+ locale = items[2]
+ if not locale: locale = 'und'
+ if not locale in settings.CITIES_LOCALES: continue
+
+ # Check if known geo id
+ geo_id = int(items[1])
+ try: geo_info = geo_id_info[geo_id]
+ except: continue
+
+ alt_type = geo_alt_names[geo_info['type']][locale]
+ alt = alt_type()
+ alt.id = items[0]
+ alt.geo = geo_info['object']
+ alt.name = items[3]
+ alt.is_preferred = items[4]
+ alt.is_short = items[5]
+ alt.save()
+ self.logger.debug(u"Added alt name: {}, {} ({})".format(locale, alt, alt.geo.name))
+
View
51 cities/management/commands/load_geonames.py
@@ -1,51 +0,0 @@
-'''
-Created on 2011-07-31
-
-@author: George
-'''
-from django.core.management.base import BaseCommand
-import os
-from django.core.management.commands.loaddata import Command as LoadCommand
-import sys
-from django.utils import simplejson
-
-def extract(country_codes):
- country_codes = [country_code.upper() for country_code in country_codes]
- country_ids = []
- region_ids = []
- city_ids = []
- fixtures_dir = '%s/../../fixtures' % os.path.dirname(__file__)
- src_data = simplejson.load(open('%s/geonames_dump.json' % fixtures_dir), 'cp1252')
- dest_data = []
- for model in src_data:
- model_type = model['model']
- if model_type == 'cities.country':
- if model['fields']['code'] in country_codes:
- country_ids.append(model['pk'])
- dest_data.append(model)
- elif model_type == 'cities.region':
- if model['fields']['country'] in country_ids:
- region_ids.append(model['pk'])
- dest_data.append(model)
- elif model_type == 'cities.city':
- if model['fields']['region'] in region_ids:
- city_ids.append(model['pk'])
- dest_data.append(model)
- elif model_type == 'cities.district':
- if model['fields']['city'] in city_ids: dest_data.append(model)
- simplejson.dump(dest_data, open('%s/extracted_data.json' % fixtures_dir, 'w'), encoding='cp1252')
-
-class Command(BaseCommand):
-
- help = 'Import selected countries into database.'
- args = "code [code ...]"
- option_list = LoadCommand.option_list
-
- def handle(self, *args, **options):
- if not len(args):
- self.stderr.write('No countries specified. Task is aborted.')
- sys.exit()
- self.stdout.write('Extracting data...\n')
- extract(args)
- labels = ('extracted_data',)
- LoadCommand().execute(*labels, **options)
View
246 cities/management/commands/update_cities_geodata.py
@@ -1,246 +0,0 @@
-"""
-GeoNames city data import script. Requires the following files:
-- http://download.geonames.org/export/dump/countryInfo.txt
-- http://download.geonames.org/export/dump/admin1CodesASCII.txt
-- http://download.geonames.org/export/dump/cities1000.zip
-
-Part of django-cities by Ben Dowling
-"""
-import os
-import sys
-import urllib
-import tempfile
-import logging
-import zipfile
-
-from django.core.management.base import BaseCommand, CommandError
-from django.db import connection
-from django.template.defaultfilters import slugify
-from django.db.models import Count
-
-from cities.models import *
-
-logging.basicConfig(level=logging.DEBUG)
-logger = logging.getLogger("cities.updater")
-
-
-def clear_data():
- logger.info("Clearing old data")
-
- cursor = connection.cursor()
- #cursor.execute("TRUNCATE TABLE `cities_city`")
- #cursor.execute("TRUNCATE TABLE `cities_country`")
- cursor.execute("TRUNCATE TABLE `cities_region`")
- cursor.execute("TRUNCATE TABLE `cities_district`")
-
-
-def import_countries():
- temp = tempfile.TemporaryFile()
-
- logger.info("Downloading countryInfo.txt")
-
- url = "http://download.geonames.org/export/dump/countryInfo.txt"
- temp.write(urllib.urlopen(url).read())
- temp.seek(0)
-
- logger.info("Parsing country data")
-
- for line in temp:
-
- if line[0] == "#":
- continue
-
- items = line.decode("utf-8").split("\t")
- country = Country()
- country.code = items[0]
- country.name = items[4]
- country.population = items[7]
- country.continent = items[8]
- country.tld = items[9][1:] # strip the leading .
-
- # Some smaller countries share a TLD. Save the one with the biggest population
- existing = Country.objects.filter(tld=country.tld)
- if existing.count():
- existing = existing[0]
- if existing.population < country.population:
- existing.delete()
- country.save()
- logger.debug(u"Replaced country %s with %s" % (existing.name, country.name))
- else:
- country.save()
- logger.debug(u"Added country %s %s" % (country.name, country.code))
-
- temp.close()
-
-def import_regions():
- temp = tempfile.TemporaryFile()
-
- logger.info("Downloading admin1codesASCII.txt")
-
- url = "http://download.geonames.org/export/dump/admin1CodesASCII.txt"
- temp.write(urllib.urlopen(url).read())
- temp.seek(0)
-
- logger.info("Parsing regions data")
-
- for line in temp: #codecs.open("admin1Codes.txt", "r", "utf-8"):
- if line[0] == "#":
- continue
-
- items = line.split("\t")
- region = Region()
- region.code = items[0]
- region.name = items[1].strip()
- region.slug = slugify(region.name)
- try:
- region.country = Country.objects.get(code=region.code[:2])
- except:
- print "Cannot find country %s - skipping" % region.code[:2]
- continue
-
- region.save()
- print "Added region %s" % (region.name,)
-
-def import_cities(data):
- for line in data: #codecs.open("cities1000.txt", "r", "utf-8"):
- if len(line) < 1 or line[0] == "#":
- continue
-
- print line
- items = line.split("\t")
- print items
- admin_type = items[11]
- type = items[7]
-
- # See http://www.geonames.org/export/codes.html
- if type in ['PPL', 'PPLA', 'PPLC', 'PPLA2', 'PPLA3', 'PPLA4'] and (type == 'PPLC' or admin_type != 'GLA'):
- city = City()
- city.id = items[0]
- city.name = items[1]
- city.slug = slugify(city.name)
- city.location = Point(float(items[4]), float(items[5]))
- city.population = items[14]
-
- region = None
- if items[11].strip():
- try:
- code = "%s.%s" % (items[8], items[11]) # Try more specific region first
- region = Region.objects.get(code=code.strip())
- except:
- pass
-
- if not region:
- try:
- code = "%s.%s" % (items[8], items[10])
- region = Region.objects.get(code=code.strip())
- except:
- print "Cannot find region %s for %s - skipping" % (code, city.name)
- continue
- city.region = region
- try:
- city.save()
- except:
- continue
- #print "Added city %s" % city
-
-def fix_regions():
- """Some large cities are placed in their own region. Fix those"""
- regions = Region.objects.annotate(count=Count('city')).filter(count=1)
- for r in regions:
- city = r.city_set.all()[0]
- try:
- nearest_cities = City.objects.filter(region__country=r.country).annotate(count=Count('region__city')).filter(count__gt=1).distance(city.location).order_by('distance')[:4] # 0 would be the same city, 1 is the nearest
- nearest_regions = {}
- for c in nearest_cities:
- nearest_regions[c.region] = 1 + nearest_regions.get(c.region, 0)
- nearest_regions = sorted(nearest_regions.iteritems(), key=lambda (k,v): (v,k))
- nearest_regions.reverse()
- nearest_region = nearest_regions[0][0]
- #print "Would move %s from %s ==> %s" % (city.name, r, nearest_region)
- city.region = nearest_region
- city.save()
- except:
- pass
-def import_districts(data):
- for line in data: #codecs.open("cities1000.txt", "r", "utf-8"):
- if len(line) < 1 or line[0] == "#":
- continue
-
- items = line.split("\t")
-
- admin_type = items[11]
- type = items[7]
-
- # See http://www.geonames.org/export/codes.html
- if type == 'PPLX' or (admin_type == 'GLA' and type != 'PPLC'):
- district = District()
- district.id = items[0]
- district.name = items[1]
- district.slug = slugify(district.name)
- district.location = Point(float(items[4]), float(items[5]))
- district.population = items[14]
- if admin_type == 'GLA':
- district.city = City.objects.filter(name='London').order_by('-population')[0] # Set city to London, UK
- else:
- district.city = City.objects.filter(population__gt=125000).distance(district.location).order_by('distance')[0] # Set the nearest city
- district.save()
- print "Added district %s" % district
-
-def cleanup():
- """ Delete all countries and regions that don't have any children, and any districts that are single children"""
-
- # Fix places in "United Kingdom (general)
- r = Region.objects.get(name='United Kingdom (general)')
- for city in r.city_set.all():
- try:
- nearest_cities = City.objects.filter(region__country=r.country).distance(city.location).exclude(region=r).order_by('distance')[:5] # 0 would be the same city, 1 is the nearest
- nearest_regions = {}
- for c in nearest_cities:
- nearest_regions[c.region] = 1 + nearest_regions.get(c.region, 0)
- nearest_regions = sorted(nearest_regions.iteritems(), key=lambda (k,v): (v,k))
- nearest_regions.reverse()
- nearest_region = nearest_regions[0][0]
- print "Moving %s to %s ==> %s" % (city.name, r, nearest_region)
- city.region = nearest_region
- city.save()
- except:
- pass
-
-
- single_districts = District.objects.annotate(count=Count('city__district')).filter(count=1)
- single_districts.delete()
-
- empty_regions = Region.objects.filter(city__isnull=True)
- empty_regions.delete()
-
- empty_countries = Country.objects.filter(region__isnull=True)
- empty_countries.delete()
-
-
-class Command(BaseCommand):
- def handle(self, *args, **options):
-
- print "Warning, this script will clear city, country, region and distinc"
- print "tables before loading updated data"
- print "Do not update if you have any models linked to this tables"
- ans = raw_input("Continue Y/N: ")
- if ans.lower() not in ["y", "yes"]:
- sys.exit()
-
- clear_data()
- import_countries()
- import_regions()
-
-
- temp = tempfile.TemporaryFile()
- logger.info("Downloading cities1000.zip")
- url = "http://download.geonames.org/export/dump/cities1000.zip"
- temp.write(urllib.urlopen(url).read())
-
- zip = zipfile.ZipFile(temp)
- data = zip.read("cities1000.txt").split("\n")
-
- import_cities(data)
- import_districts(data)
- fix_regions()
- cleanup()
View
187 cities/models.py
@@ -1,83 +1,122 @@
+from django.conf import settings
+from django.utils.encoding import force_unicode
from django.contrib.gis.db import models
from django.contrib.gis.geos import Point
-
+from util import create_model, un_camel
+
class Country(models.Model):
- name = models.CharField(max_length = 200)
- code = models.CharField(max_length = 2, db_index=True)
- population = models.IntegerField()
- continent = models.CharField(max_length = 2)
- tld = models.CharField(max_length = 5, unique=True)
-
- objects = models.GeoManager()
-
- def __unicode__(self):
- return self.name
-
- @property
- def hierarchy(self):
- return [self]
-
- class Meta:
- ordering = ['name']
+ name = models.CharField(max_length=200, db_index=True) # ASCII name
+ slug = models.CharField(max_length=200)
+ code = models.CharField(max_length=2, db_index=True)
+ population = models.IntegerField()
+ continent = models.CharField(max_length=2)
+ tld = models.CharField(max_length=5)
+ objects = models.GeoManager()
+
+ class Meta:
+ ordering = ['name']
+ verbose_name_plural = "countries"
+
+ def __unicode__(self):
+ return self.name
+
+ @property
+ def hierarchy(self):
+ return [self]
class Region(models.Model):
- name = models.CharField(max_length = 200)
- slug = models.CharField(max_length = 200, db_index=True)
- code = models.CharField(max_length = 10, db_index=True)
- country = models.ForeignKey(Country)
- objects = models.GeoManager()
-
- def __unicode__(self):
- return "%s, %s" % (self.name, self.country)
-
- @property
- def hierarchy(self):
- list = self.country.hierarchy
- list.append(self)
- return list
-
-class CityManager(models.GeoManager):
- def nearest_to(self, lat, lon):
- #Wrong x y order
- #p = Point(float(lat), float(lon))
- p = Point(float(lon), float(lat))
- return self.nearest_to_point(p)
-
- def nearest_to_point(self, point):
- return self.distance(point).order_by('distance')[0]
+ name = models.CharField(max_length=200, db_index=True) # ASCII name
+ name_std = models.CharField(max_length=200, db_index=True) # UTF-8 international standard name
+ slug = models.CharField(max_length=200)
+ code = models.CharField(max_length=10, db_index=True)
+ country = models.ForeignKey(Country)
+ objects = models.GeoManager()
+
+ def __unicode__(self):
+ return u'{}, {}'.format(force_unicode(self.name_std), self.country)
+
+ @property
+ def hierarchy(self):
+ list = self.country.hierarchy
+ list.append(self)
+ return list
class City(models.Model):
- name = models.CharField(max_length = 200)
- slug = models.CharField(max_length = 200, db_index=True)
- region = models.ForeignKey(Region)
- location = models.PointField()
- population = models.IntegerField()
-
- objects = CityManager()
-
- def __unicode__(self):
- return "%s, %s" % (self.name, self.region)
-
- @property
- def hierarchy(self):
- list = self.region.hierarchy
- list.append(self)
- return list
+ name = models.CharField(max_length=200, db_index=True) # ASCII name
+ name_std = models.CharField(max_length=200, db_index=True) # UTF-8 international standard name
+ slug = models.CharField(max_length=200)
+ region = models.ForeignKey(Region)
+ location = models.PointField()
+ population = models.IntegerField()
+ objects = models.GeoManager()
+
+ class Meta:
+ verbose_name_plural = "cities"
+
+ def __unicode__(self):
+ return u'{}, {}'.format(force_unicode(self.name_std), self.region)
+
+ @property
+ def hierarchy(self):
+ list = self.region.hierarchy
+ list.append(self)
+ return list
class District(models.Model):
- name = models.CharField(max_length = 200)
- slug = models.CharField(max_length = 200, db_index=True)
- city = models.ForeignKey(City)
- location = models.PointField()
- population = models.IntegerField()
-
- objects = models.GeoManager()
-
- def __unicode__(self):
- return u"%s, %s" % (self.name, self.city)
-
- @property
- def hierarchy(self):
- list = self.city.hierarchy
- list.append(self)
- return list
+ name = models.CharField(max_length=200, db_index=True) # ASCII name
+ name_std = models.CharField(max_length=200, db_index=True) # UTF-8 international standard name
+ slug = models.CharField(max_length=200)
+ city = models.ForeignKey(City)
+ location = models.PointField()
+ population = models.IntegerField()
+ objects = models.GeoManager()
+
+ def __unicode__(self):
+ return u'{}, {}'.format(force_unicode(self.name_std), self.city)
+
+ @property
+ def hierarchy(self):
+ list = self.city.hierarchy
+ list.append(self)
+ return list
+
+class GeoAltNameManager(models.GeoManager):
+ def get_preferred(self, default=None, **kwargs):
+ """
+ If multiple names are available, get the preferred, otherwise return any existing or the default.
+ Extra keywords can be provided to further filter the names.
+ """
+ try: return self.get(is_preferred=True, **kwargs)
+ except self.model.DoesNotExist:
+ try: return self.filter(**kwargs)[0]
+ except IndexError: return default
+
+def _create_geo_alt_name(geo_type):
+ geo_alt_names = {}
+ for locale in settings.CITIES_LOCALES:
+ name_format = geo_type.__name__ + '{}' + locale.capitalize()
+ name = name_format.format('AltName')
+ geo_alt_names[locale] = create_model(
+ name = name,
+ fields = {
+ 'geo': models.ForeignKey(geo_type, # Related geo type
+ related_name = 'alt_names_' + locale),
+ 'name': models.CharField(max_length=200, db_index=True), # Alternate name
+ 'is_preferred': models.BooleanField(), # True if this alternate name is an official / preferred name
+ 'is_short': models.BooleanField(), # True if this is a short name like 'California' for 'State of California'
+ 'objects': GeoAltNameManager(),
+ '__unicode__': lambda self: force_unicode(self.name),
+ },
+ app_label = 'cities',
+ module = 'cities.models',
+ options = {
+ 'db_table': 'cities_' + un_camel(name),
+ 'verbose_name': un_camel(name).replace('_', ' '),
+ 'verbose_name_plural': un_camel(name_format.format('AltNames')).replace('_', ' '),
+ },
+ )
+ return geo_alt_names
+
+geo_alt_names = {}
+for type in [Country, Region, City, District]:
+ geo_alt_names[type] = _create_geo_alt_name(type)
View
51 cities/util.py
@@ -0,0 +1,51 @@
+import re
+from django.db import models
+from django.contrib import admin
+
+first_cap_re = re.compile('(.)([A-Z][a-z]+)')
+all_cap_re = re.compile('([a-z0-9])([A-Z])')
+def un_camel(name):
+ """
+ Convert CamelCase to camel_case
+ """
+ s1 = first_cap_re.sub(r'\1_\2', name)
+ return all_cap_re.sub(r'\1_\2', s1).lower()
+
+
+def create_model(name, fields=None, app_label='', module='', options=None, admin_opts=None):
+ """
+ Dynamically create model specified with args
+ """
+ class Meta:
+ # Using type('Meta', ...) gives a dictproxy error during model creation
+ pass
+
+ if app_label:
+ # app_label must be set using the Meta inner class
+ setattr(Meta, 'app_label', app_label)
+
+ # Update Meta with any options that were provided
+ if options is not None:
+ for key, value in options.iteritems():
+ setattr(Meta, key, value)
+
+ # Set up a dictionary to simulate declarations within a class
+ attrs = {'__module__': module, 'Meta': Meta}
+
+ # Add in any fields that were provided
+ if fields:
+ attrs.update(fields)
+
+ # Create the class, which automatically triggers ModelBase processing
+ model = type(name, (models.Model,), attrs)
+
+ # Create an Admin class if admin options were provided
+ if admin_opts is not None:
+ class Admin(admin.ModelAdmin):
+ pass
+ for key, value in admin_opts:
+ setattr(Admin, key, value)
+ admin.site.register(model, Admin)
+
+ return model
+
View
4 setup.py
@@ -1,10 +1,10 @@
from setuptools import setup, find_packages
-setup(name='cities',
+setup(name='django-cities',
version='0.1',
description='Django Cities',
author='Ben Dowling',
packages=find_packages(),
include_package_data=True,
zip_safe=False,
- )
+ )
Please sign in to comment.
Something went wrong with that request. Please try again.