Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
|commonlib @ bb08a37|
MaPit ===== pylib/mapit is a standard GeoDjango project; the rest is mostly to fit within mySociety's standard deployment. Checking out the code --------------------- A standard git clone will get you the repository. Some code (used by multiple mySociety projects) is kept in a submodule repository called commonlib - to get that, simply type "git submodule update --init" in the main repository directory and it'll appear. Installation ------------ Our version is running on Debian lenny, and we've successfully had it running on Ubuntu 10.04 as well - to install PostGIS and GeoDjango I just followed all the instructions (including creating the template) at http://docs.djangoproject.com/en/dev/ref/contrib/gis/install/ However, not only is the PostGIS that is installed missing SRID 900913, as the GeoDjango docs tell you, but both SRID 27700 (British National Grid) and SRID 29902 (Irish National Grid) are incorrect (and they're quite important for this application!). After you've installed and got a PostGIS template, log in to it and update the proj4text column of SRID 27700 to include +datum=OSGB36, and update SRID 29902 to have +datum=ire65. (This may not be necessary, depending on your version of PostGIS, but do check.) Initial instructions for all locations -------------------------------------- 1. Make sure OPTION_STAGING is 0 in conf/general, otherwise you'll run out of memory as it tries to remember all the SQL queries made. 2. Optional: turn off escape_string_warning in Postgres' config (unless you want your server to run out of disc space logging all the errors, as ours did the first time I imported overnight and didn't realise). United Kingdom -------------- Here are the basic instructions to install OS OpenData and NSPD Open: 1. Set OPTION_AREA_SRID in conf/general to 27700 (as Boundary-Line shapes are in OSGB). 2. Download the latest Code-Point Open, Boundary-Line and NSPD from <http://parlvid.mysociety.org:81/os/>, and save/unzip in the data directory. 3. Change to the pylib/mapit directory, create database tables and start the site: ./manage.py syncdb ./manage.py runserver # (Alternatively, set up a live web server however you wish.) 4. Run the following: ./manage.py generation_create --commit --desc "Initial import." ./manage.py import_boundary_line --control=controls.first-gss --commit `ls ../../data/Boundary-Line/*.shp|grep -v high_water` # (You can run without --commit to do a dry run.) # first-gss in the above assumes the Boundary Line you're importing is # October 2010 or later, and uses the new GSS codes. ./manage.py find_parents ./manage.py import_codepoint ../../data/Code-Point-Open/*.csv ./manage.py scilly ../../data/Code-Point-Open/tr.csv ./manage.py import_nspd_ni_areas ./manage.py import_nspd_ni ../../data/ONSPD.csv ./manage.py import_nspd_crown_dependencies ../../data/ONSPD.csv ./manage.py generation_activate --commit For notes on what was done to create generations as you can see on mapit.mysociety.org, see the end of this file. Notes on future releases ------------------------ When a new Code-Point is released, you should just be able to run import_codepoint and scilly; when new NSPD is out, import_nspd_ni if it's only postcodes that have changed, or import_nspd_ni_areas first if boundary changes too (this is incomplete, it doesn't use a control file like import_boundary_line does); when new Boundary-Line, import_boundary_line and find_parents. In May 2011, the Northern Ireland Assembly boundaries move to match the current Parliamentary boundaries - import_nspd_ni_areas needs changing to cope with that, it currently only creates the current (pre May 2011) boundaries. You can manually increase the generation_high_id when something is new and something else isn't (most commonly, a new Boundary-Line means a new generation for Great Britain, and you can then increase the Northern Ireland boundaries manually to be in the new generation). Norway ------ Here are the basic instructions to install data from OSM: 1. Set OPTION_AREA_SRID in conf/general to 4326 (as OSM shapes are in WGS84). 2. cd bin and run "python osm_to_kml" to fetch OSM XML and convert it to KML. 3. Change to the pylib/mapit directory, create database tables and start the site: ./manage.py syncdb ./manage.py runserver 4. Run the following (you can run anything without --commit to do a dry run): ./manage.py generation_create --commit --desc "Initial import." ./manage.py import_norway_osm --commit ../../data/cache/*.kml ./manage.py generation_activate --commit Please see below for information on where osm_to_kml gets its OSM data from. Alternatively, here are the basic instructions to install the N5000 data: 1. Set OPTION_AREA_SRID in conf/general to 4326 (as we'll put N5000 shapes into WGS84). 2. Download N5000 from <http://www.statkart.no/nor/Land/Kart_og_produkter/N5000_-_gratis_oversiktskart/> and save/unzip in the data directory. 3. Change to the pylib/mapit directory, create database tables and start the site: ./manage.py syncdb ./manage.py runserver 4. Run the following (you can run anything without --commit to do a dry run): ./manage.py generation_create --commit --desc "Initial import." ./manage.py import_norway_n5000 --commit ../../data/N5000\ shape/N5000_AdministrativFlate.shp ./manage.py generation_activate --commit You should now be able to go to /point/4326/10.756389,59.949444 and have Oslo returned as a result. Norway OSM data --------------- The way osm_to_kml works is to fetch a number of fixed and pre-defined relation IDs from OSM - one (412437) containing all fylke, and then one for each fylke containing all the kommune inside. These relations should stay and (now they're correct) not change within OpenStreetmap, though of course the underlying relations can have their boundaries improved and so on. See the relation_ids list in the source of bin/osm_to_kml if you'd like to see the other relation IDs. The OSM tags 'name', and 'name:no' if 'name' is not set, are used to find the primary name of the fylke and kommune. In addition, the values of the tags 'name:smi', 'name:fi' are imported into mapit. Only the primary name is shown in the mapit web pages and JSON data, while the other names are stored in the database. The kommune and fylke number (ID) is fetched from a the tag 'ref' in OSM, and if it is missing a static list of such IDS in mapit/data/norway/ids.csv is consulted using the name (for fylke) or name+fylke (for kommune) as the key. Improvements / patches ---------------------- Are welcome :) ATB, Matthew ===================================== Notes on creating what's live ----------------------------- When creating what you see at mapit.mysociety.org, to enable it to have pre-2010 election boundaries, I ran the above (or rather, what existed at the time, which is not identical) twice, once with 2009-10 Boundary-Line and then the 2010-05 edition. I had to write the 2010-05 control file you can see, did not re-run import_codepoint (as no postcodes had changed), and only ran the NI stuff the second generation (as we only had current data). The commands I basically ran are below. Even worse, as I had to maintain IDs between our old and new versions of mapit, I then matched up all IDs and names using the scripts in bin, manually inserted some generation 10 areas (in data) for FixMyStreet and some generation 12 NI WMC areas for WriteToThem, and manually added our test/fake areas that used to be in code but can now happily sit in the database along with everything else. You probably don't need any of that for your own install. # Create inactive generation. ./manage.py import_boundary_line --control=controls.2009-10 `ls ../../data/Boundary-Line/2009-10/*.shp|grep -v high_water` ./manage.py import_codepoint ../../data/Code-Point-Open-2010-05/*.csv ./manage.py find_parents # Not importing NI here, as that only has the current boundaries. ./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv # Make generation active, add another inactive generation ./manage.py import_boundary_line --control=controls.2010-05 `ls ../../data/Boundary-Line/2010-05/*.shp|grep -v high_water` # import_codepoint not needed as it's the same and there's no P-in-P tests! ./manage.py find_parents ./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv # I doubt the boundaries change! But updates the generation. ./manage.py import_nspd ../../data/NSPD-2010-05.csv # This is now split into two scripts, see below. # Make generation active.