Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Placeholder for some ideas about OpenStreetMap extracts

branch: master
Octocat-spinner-32 coastline-preview Added complete(ish) upload shellscript December 15, 2011
Octocat-spinner-32 previews Added Bucharest, Romania by request December 10, 2013
Octocat-spinner-32 Makefile new area calculation, pretty print area output March 29, 2013
Octocat-spinner-32 README.md Corrected AMI and instance type in README March 25, 2013
Octocat-spinner-32 bbmap.js Cleaned up part of bbmap.js that was unused March 26, 2013
Octocat-spinner-32 build-coastshapes-script.py Compressed coastshapes.sh to fit within dpaste.com limits February 20, 2012
Octocat-spinner-32 build-index.py Switched back to older method of number formatting. April 14, 2013
Octocat-spinner-32 build-osm2pgsql-script.py Small path updates to extract.sh and osm2pgsql.sh March 24, 2013
Octocat-spinner-32 build-osmosis-script.py Updated osmosis.sh with continent groups March 22, 2013
Octocat-spinner-32 cities.txt Added Bucharest, Romania by request December 10, 2013
Octocat-spinner-32 coastline-errors.sh I guess these are in a different place now? December 08, 2013
Octocat-spinner-32 coastshapes.sh Added Bucharest, Romania by request December 10, 2013
Octocat-spinner-32 compose-city-previews.py Updated design of index page with new CSS and new Mapquest Open style… March 24, 2013
Octocat-spinner-32 extract.sh Small path updates to extract.sh and osm2pgsql.sh March 24, 2013
Octocat-spinner-32 index.html Updated index.html with 2013-07-01 extracts July 10, 2013
Octocat-spinner-32 kill-self.sh Added --no-kill option to run-instance.py January 29, 2012
Octocat-spinner-32 lib.py check in missing file March 29, 2013
Octocat-spinner-32 osm2pgsql.sh Added Bucharest, Romania by request December 10, 2013
Octocat-spinner-32 osm2pgsql.style Removed osm2pgsql tags that were too long for postgres column names February 10, 2012
Octocat-spinner-32 osmosis.sh Added Bucharest, Romania by request December 10, 2013
Octocat-spinner-32 run-instance.py Updated run script with AMI and instance type we actually want to use March 22, 2013
Octocat-spinner-32 style.css override Leaflet popup style March 26, 2013
Octocat-spinner-32 upload-files.sh Made file upload at the end an option but left default behavior on February 15, 2012
README.md

Extractotron

In one of my SotM 2011 talks, I suggested that OSM should have regular extracts of metropolitan areas for major world cities and their surrounding areas, regardless of state or country. This is a repository for a bit of code to make that real. I’ve decided to use Amazon S3 to host files, and EC2 to do the extracting. extract.sh is a script that performs the actual work with Osmosis and terminates its host machine upon completion.

Adding New Cities

Extractotron is run a few times per month, to http://metro.teczno.com. If you have cities to add, you can add them directly to cities.txt and send me a pull request in Github.

cities.txt is a tab-delimited file, and has eight fields:

  • group

    One of "Africa", "Asia", "Europe", "Middle East", "North America", "South America", or Oceania.

  • geonameid

    It's nice to have a link back to the Geonames dataset.

  • top, left, bottom, right

    Geographic bounding box of the metropolitan area. These should be larger rather than smaller to surrounding countryside wherever possible. For example of coverage area, see Moscow.

  • slug

    Short, lowercase version of the name with dashes for spaces that's used for files.

  • name

    Full name of the city.

Rolling Your Own

To use Extractotron yourself, use run-instance.py:

python run-instance.py <your AWS access key> <your AWS secret> <your bucket name>

By default, run-instance.py instantiates an m2.4xlarge instance with Alestic's Ubuntu 10.04 Lucid instance-store AMI ami-bffa6fd6. You can change the instance type or AMI, see python run-instance.py --help for details.

Currently, the cities list is very short. Help me expand it by modifying cities.txt and sending a pull request via Github.

Code Documentation

The Makefile generates various derived files from the cities.txt file. These scripts are utilities to do the extracts, generate utility scripts, JPG previews, and an HTML index based on the regions specified in index.html. Note the result of this Makefile is checked in at github so there's no immediate need to run make on a fresh clone of the repo.

  • build-osmosis-script.py

    Generate the shell script osmosis.sh which uses Osmosis to extract subsets of OSM based on the bounding boxes in cities.txt

  • build-osm2pgsql-script.py

    Generate the shell script osm2pgsql.sh which uses imposm to import generated OSM metro subsets into a PostGIS database.

  • build-coastshapes-script.py

    Generate the shell script coastshapes.sh which uses ogr2ogr to generate coastline shapefiles.

  • build-index.py

    Generate index.html, the current HTML interface to allow download of various extracted OSM data.

  • compose-city-previews.py

    Generate the JPG images in previews, snapshot maps of the various metros being extracted.

The scripts generated above are then used by Extractotron to generate the actual extracts. The code for doing the extracts includes

  • run-instance.py

    Utility script to create an EC2 instance to run the extract. See above for details.

  • extract.sh

    Template for the shell script to do the extraction. run-instance.py substitutes pathnames for the scripts generated in the Makefile so that EC2 can then run the extraction. See also upload-files.sh and kill-self.sh.

Something went wrong with that request. Please try again.