Getting started: DATA
Clone this wiki locally
Before you begin
Sources of Data
- Detailed neighborhood streets, highways, rail, paths, and points-of-interest (POI) data comes from OpenStreetMap.
- General world and regional scale map data such as countries or states comes from Natural Earth.
- Detailed place names come from GeoNames, which collate them from a variety of sources. USGS and US Census are also good sources.
- Digital elevation, landcover and similar data comes from US Geological Survey.
- US statistical divisions like metro areas or counties come from US Census.
The first three are global in scope, the last two are mostly US-only, alternatives exist on a per-country basis.
Often data is pre-process to prepare it for rendering. Sometimes to make it look a certain way, other times to make it run faster.
- Does your basemap need to reflect the latest edits?
- Do you cover a rapidly expanding urban area where the roads change all the time and you're users will notice?
- Will you re-render your basemap on a never, yearly, monthly, weekly, daily, frequency?
- World up to a certain zoom then just your local geography?
Open Street Map
Metro Extracts - Get OSM data extracts for big cities world wide in small chunks (mb instead of gb). Formats include XML, PBF, and osm2pgsql + imposm shapefiles. Includes coastline shapefile option. Updated monthly.
BBBike extracts - Allows you to extracts areas from Planet.osm in OSM, PBF, Garmin, Osmand or ESRI shapefile format. The maximum area size is 16,000,000 square km, or 768MB file size. It takes between 10-30 minutes to extract an area. You will be notified by e-mail if your extract is ready for download.
WeoGeo OSM Extracts - Free download from their marketplace. Data from the WeoGeo OSM listing can currently be output as FGDB, KML, DWG, FFS, and CSV (no geo info is included). Their developers are working on SHP and TAB support.
Full OSM Planet
The big kahuna. You'll need a beefy machine. Updated daily.
For making detailed city maps from zooms 12 to 19+.
Note: OSM redaction started in July 2012. Once it is complete, OSM will be ready to distribute data under the ODbL and they’ll advise of that with a separate announcement. The final pre-redaction dataset available under CC-BY-SA has now been generated at http://planet.openstreetmap.org/planet-120704.osm.bz2.
Osmosis - Parsing and merging large Planet or other XML and binary OpenStreetMap data files.
impOSM - Optimize your OSM files per zoom for faster, for performant rendering.
imposm -m imposm-mapping.py -U <user_name> -d <db_name> --proj=EPSG:900913 -c 4 --read --write --deploy-production-tables osm-planet-test-areas.pbf
imposm -c 3 -U <user_name> -d <db_name> -m <mapping_file.py> --overwrite-cache --read --write --deploy-production-tables osm-planet-test-areas.pbf osm-planet-test-areas.pbf
osmfilter - Similar.
osm2pgsql - Get raw OSM files into PostGIS databases.
High Road - Add views into the data, grouping ways into large, medium, and small roads per zoom.
OSM Data Dictionary
- https://github.com/nvkelso/geo-how-to/wiki/OpenStreetMap - Links to general how-tos for OSM.
- High Road - Important to drop and then readd this on each OSM import. Details on the High Road page.
- NaturalEarthData.com - For making generalized world maps from zooms 0 to 8.
Help getting Natural Earth into PostGIS
- Aaron's Naturalearth-tools - Shell script for fetching the data, clean-geometry, and reprojecting to web Mercator from Aaron
Spherical mercator extent and projection (ref):
EXTENT="-180 -85.05112878 180 85.05112878" P900913="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs"
Use http://trac.osgeo.org/gdal/wiki/ConfigOptions#OGR_ENABLE_PARTIAL_REPROJECTION and clip source to include only data within spherical mercator world square. Encoding conversion will only work as of GDAL 1.9.x. Also note -segmentize 1 adds vertex every 1 degree distance on lines for better reprojection fidelity.
ogr2ogr \ --config OGR_ENABLE_PARTIAL_REPROJECTION TRUE --config SHAPE_ENCODING WINDOWS-1252 \ -t_srs "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 \ +units=m +nadgrids=@null +no_defs" -lco ENCODING=UTF-8 \ -clipsrc -180 -85.05112878 180 85.05112878 -segmentize 1 -skipfailures \ outfile.shp infile.shp
- Often you can get GIS map data from your local or state government.
- From PDF data to map »