Miami-Dade County Address + Building (+POI?) Import
Software tools and technical description of the Miami-Dade OSM import in 2018
Largely based on the 2016 Large Building Import. Tool/documentation still under construction.
PostGIS on your system. You can find some help here.
You will also need the
requests python packages.
Install osmosis with
apt-get install osmosis on Ubuntu/Debian (other platforms see http://wiki.openstreetmap.org/wiki/Osmosis/Installation).
GDAL/OGR for your system. Used for importing shapefiles to Postgres with
Download the data files from here:
Building Footprint 2D - http://gis-mdc.opendata.arcgis.com/datasets/building-footprint-2d Address with condo - http://gis-mdc.opendata.arcgis.com/datasets/address-with-condo?geometry=-82.952%2C25.187%2C-77.854%2C26.054 Census Tracts - http://gis-mdc.opendata.arcgis.com/datasets/tract-2010
- Create a PostgreSQL database and make sure user 'postgres' with the password 'postgres' has access to it. Until the code is cleaned,
osm_testis hardcoded in some cases. We will make the code base f more flexible so that it can be easily used for other imports.
- Set up the DB (extensions, tables)
python data_prep/main.py --setup
Get the data
- Import shapefiles to db. (I store the shapefiles in a folder called data. Pass it as the first argument) to the following shell script. This step uses
ogr2ogrto populate the
building_footprint_2dtables in Postgres.
The following steps import current OSM data to the same database.
- Grab OSM buildings from OverpassAPI (store them in osm_buildings table)
python data_prep/main.py --buildings_download
- Grab OSM Addresses from OverpassAPI (osm_addresses table)
python data_prep/main.py --address_download
- Grab OSM highways/railways from OverpassAPI (osm_highway_railway)
python data_prep/main.py --roads_download
Prepare the data for processing/conversion
- Get rid of duplicate addresses in the import dataset (i.e. condo units).
python data_prep/main.py --dedupe_address
- Add some extra fields to tables.
python data_prep/main.py --add_fields
- Convert closed linestrings from Overpass to Polygon
python data_prep/main.py --convert_poly
- Create spatial indexes to ensure that spatial queries are effective. Update databaes statistics.
python data_prep/main.py --index_data python data_prep/main.py --vacuum
- Check import addresses against OSM adddresses. Flag those in the proximity of existing addresses and exclude them.
python data_prep/main.py --check_address
- Check import buildings against OSM buildings. Flag those intersecting with existing buildings and exlude them.
python data_prep/main.py --check_building
- Check buildings crossing highway and railway features. Add
fixme=highway/railway crosses building
python data_prep/main.py --check_road_rail
- Assign address to buildings where there is only one overlapping address point (or there is only one within 5 meters)
python data_prep/main.py --assign_address
- Get rid of unnecessary nodes using the Douglas-Peucker algorithm (tolerance: 0.3 meters)
python data_prep/main.py --simplify
ogr2osmin parent directory
cd .. git clone --recursive https://github.com/pnorman/ogr2osm
- Navigate back to the project folder
- Generate test datasets. Data files for all tract polygons: use
cd data_conversion ./generate_osm_files.sh test