A web service to map postcodes to administrative boundaries and more
Fetching latest commit…
Cannot retrieve the latest commit at this time
|LICENSE.txt||Initial licence file.|
MapIt ===== The mapit directory is a standard GeoDjango app, and the project directory is an example GeoDjango project using that app. Other top-level things are mostly to fit within mySociety's standard deployment, or country specific data, one-off scripts and so on. MapIt has been installed on Debian lenny and squeeze, and on Ubuntu from 10.04 onwards. If GeoDjango and PostGIS run on it, it should theoretically be okay; do let us know of any problems. Commands are intended to be run via the terminal or over ssh. Installation ------------ MapIt currently uses Postgres/PostGIS as its database backend - there's no reason why e.g. SpatiaLite could not be used successfully, but it has never been tried. To install GeoDjango and PostGIS, please follow all the standard instructions (including creating the template) at: http://docs.djangoproject.com/en/dev/ref/contrib/gis/install/#ubuntudebian And anything else you need to set up Django as normal. [ Note for UK/Ireland: Not only is the PostGIS that is installed missing SRID 900913, as the GeoDjango docs tell you, but both SRID 27700 (British National Grid) and SRID 29902 (Irish National Grid) can be incorrect (and they're quite important for this application!). After you've installed and got a PostGIS template, log in to it and update the proj4text column of SRID 27700 to include +datum=OSGB36, and update SRID 29902 to have +datum=ire65. This may not be necessary, depending on your version of PostGIS, but do check. ] You will also need a couple of other Debian packages, so install them: sudo apt-get install python-yaml memcached python-memcache git-core You will also need to be using South to manage schema migrations. Installation as a Django app ---------------------------- As mapit is a Django app, if you are making a GeoDjango project you can simply use mapit from within your project like any normal Django app: * Make sure it is on sys.path (a packaged install e.g. with 'pip install django-mapit' does this for you); * Add 'mapit' and 'django.contrib.gis' to your INSTALLED_APPS; * Add the following settings to settings.py: MAPIT_AREA_SRID - the SRID of your area data (if in doubt, set to 4326) MAPIT_COUNTRY - used to supply country specific functions (such as postcode validation). If you look at the country files in mapit/countries/ you can see how to add specialised country-specific functions. MAPIT_RATE_LIMIT - a list of IP addresses or User Agents excluded from rate limiting * Set up a path in your main urls.py to point at mapit.urls. * run './manage.py syncdb' and './manage.py migrate' to ensure the db is set up This approach is new, so please let us know if something goes wrong or could be improved. Installation standalone with the example project ------------------------------------------------ A standard git clone will get you the repository: git clone git://github.com/mysociety/mapit.git Now in a terminal, navigate to the mapit directory you've just cloned. Copy conf/general.yml-example to conf/general.yml and edit it to point to your local postgresql database, and edit the other settings as per the documentation given in that file. If you don't know what SRID to use, delete that line or set it to 4326. COUNTRY is used for e.g. differing sorts of postcode (zipcode) validation - if you look at the country files in mapit/countries/ you can see how to add specialised country-specific functions to validate postcodes etc. If you're going to be importing big datasets, make sure that DEBUG is False; otherwise, you'll run out of memory as it tries to remember all the SQL queries made. Optionally, also turn off escape_string_warning in Postgres' config (unless you want your server to run out of disc space logging all the errors, as ours did the first time I imported overnight and didn't realise). At this stage, you should be able to set up the database and run the development server. Do add an admin user when prompted: cd project ./manage.py syncdb ./manage.py migrate mapit ./manage.py runserver (Alternatively, set up a live web server however you wish - see the Deployment Django documentation for details beyond the scope of this document.) You can then visit http://localhost:8000/ and hopefully see the default MapIt homepage. http://localhost:8000/admin/ should show the admin interface. This is enough to have a working site. You can create areas and postcodes using the admin interface and they will automatically work in the API-based front end. However, if you have some bulk data you wish to import (which will make things easier), you will need to import this data into MapIt. Below are some instructions for such bulk import for the United Kingdom and Norway, which should show the general format for how it's done. United Kingdom -------------- Here are the basic instructions to install OS OpenData and ONSPD: 1. AREA_SRID in conf/general.yml should be 27700 (as Boundary-Line shapes are in OSGB). 2. Download the latest Code-Point Open, Boundary-Line and ONSPD from <http://parlvid.mysociety.org:81/os/>, and save/unzip in the data directory. 3. Change to the project directory, and create database tables (if you haven't already done this): ./manage.py syncdb ./manage.py migrate mapit 4. Run the following in order: ./manage.py generation_create --commit --desc "Initial import." ./manage.py loaddata uk ./manage.py import_boundary_line --control=mapit.controls.first-gss --commit `ls ../data/Boundary-Line/Data/*.shp|grep -v high_water` # (You can run without --commit to do a dry run.) # first-gss in the above assumes the Boundary Line you're importing is # October 2010 or later, and uses the new GSS codes. ./manage.py find_parents ./manage.py import_codepoint ../data/Code-Point-Open/*.csv ./manage.py scilly ../data/Code-Point-Open/tr.csv ./manage.py import_nspd_ni_areas ./manage.py import_nspd_ni ../data/ONSPD.csv ./manage.py import_nspd_crown_dependencies ../data/ONSPD.csv ./manage.py generation_activate --commit For notes on what was done to create generations as you can see on mapit.mysociety.org, see the end of this file. Notes on future releases ------------------------ When a new Code-Point is released, you should just be able to run import_codepoint and scilly; when new ONSPD is out, import_nspd_ni if it's only postcodes that have changed, or import_nspd_ni_areas first if boundary changes too (this is incomplete, it doesn't use a control file like import_boundary_line does); when new Boundary-Line, import_boundary_line and find_parents. In May 2011, the Northern Ireland Assembly boundaries move to match the current Parliamentary boundaries - import_nspd_ni_areas needs changing to cope with that, it currently only creates the current (pre May 2011) boundaries. You can manually increase the generation_high_id when something is new and something else isn't (most commonly, a new Boundary-Line means a new generation for Great Britain, and you can then increase the Northern Ireland boundaries manually to be in the new generation). Norway ------ Here are the basic instructions to install data from OSM: 1. Set AREA_SRID in conf/general.yml to 4326 (as OSM shapes are in WGS84). 2. cd bin and run "python osm_to_kml" to fetch OSM XML and convert it to KML. 3. Change to the project directory, and create database tables: ./manage.py syncdb ./manage.py migrate mapit 4. Run the following (you can run anything without --commit to do a dry run): ./manage.py generation_create --commit --desc "Initial import." ./manage.py loaddata norway ./manage.py import_norway_osm --commit ../../data/cache/*.kml ./manage.py import_area_unions --commit data/norway/regions.csv ./manage.py generation_activate --commit Please see below for information on where osm_to_kml gets its OSM data from. Alternatively, here are the basic instructions to install the N5000 data: 1. Set AREA_SRID in conf/general.yml to 4326 (as we'll put N5000 shapes into WGS84). 2. Download N5000 from <http://www.statkart.no/nor/Land/Kart_og_produkter/N5000_-_gratis_oversiktskart/> and save/unzip in the data directory. 3. Change to the project directory, and create database tables: ./manage.py syncdb ./manage.py migrate mapit 4. Run the following (you can run anything without --commit to do a dry run): ./manage.py generation_create --commit --desc "Initial import." ./manage.py loaddata norway ./manage.py import_norway_n5000 --commit ../../data/N5000\ shape/N5000_AdministrativFlate.shp ./manage.py import_area_unions --commit data/norway/regions.csv ./manage.py generation_activate --commit You should now be able to go to /point/4326/10.756389,59.949444 and have Oslo returned as a result. Norway OSM data --------------- The way osm_to_kml works is to fetch a number of fixed and pre-defined relation IDs from OSM - one (412437) containing all fylke, and then one for each fylke containing all the kommune inside. These relations should stay and (now they're correct) not change within OpenStreetmap, though of course the underlying relations can have their boundaries improved and so on. See the relation_ids list in the source of bin/osm_to_kml if you'd like to see the other relation IDs. The OSM tags 'name', and 'name:no' if 'name' is not set, are used to find the primary name of the fylke and kommune. In addition, the values of the tags 'name:smi', 'name:fi' are imported into mapit. Only the primary name is shown in the mapit web pages and JSON data, while the other names are stored in the database. The kommune and fylke number (ID) is fetched from a the tag 'ref' in OSM, and if it is missing a static list of such IDS in mapit/data/norway/ids.csv is consulted using the name (for fylke) or name+fylke (for kommune) as the key. Improvements / patches ---------------------- Are welcome :) ATB, Matthew ===================================== Notes on creating what's live ----------------------------- When creating what you see at mapit.mysociety.org, to enable it to have pre-2010 election boundaries, I ran the above (or rather, what existed at the time, which is not identical) twice, once with 2009-10 Boundary-Line and then the 2010-05 edition. I had to write the 2010-05 control file you can see, did not re-run import_codepoint (as no postcodes had changed), and only ran the NI stuff the second generation (as we only had current data). The commands I basically ran are below. Even worse, as I had to maintain IDs between our old and new versions of mapit, I then matched up all IDs and names using the scripts in bin, manually inserted some generation 10 areas (in data) for FixMyStreet and some generation 12 NI WMC areas for WriteToThem, and manually added our test/fake areas that used to be in code but can now happily sit in the database along with everything else. You probably don't need any of that for your own install. # Create inactive generation. ./manage.py import_boundary_line --control=mapit.controls.2009-10 `ls ../../data/Boundary-Line/2009-10/*.shp|grep -v high_water` ./manage.py import_codepoint ../../data/Code-Point-Open-2010-05/*.csv ./manage.py find_parents # Not importing NI here, as that only has the current boundaries. ./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv # Make generation active, add another inactive generation ./manage.py import_boundary_line --control=mapit.controls.2010-05 `ls ../../data/Boundary-Line/2010-05/*.shp|grep -v high_water` # import_codepoint not needed as it's the same and there's no P-in-P tests! ./manage.py find_parents ./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv # I doubt the boundaries change! But updates the generation. ./manage.py import_nspd ../../data/NSPD-2010-05.csv # This is now split into two scripts, see below. # Make generation active.