Mapzen Vector Tile Service
Clone this wiki locally
Mapzen Vector Tiles
See info about the hosted version of this service here:
(tested under Ubuntu 14.04)
# install misc tools sudo apt-get install git # install postgres / postgis sudo apt-get install postgresql postgresql-contrib postgis postgresql-9.3-postgis-2.1 # Install jinja2 sudo apt-get install python-jinja2 # install tilezen fork of osm2pgsql sudo apt-add-repository ppa:tilezen sudo apt-get update sudo apt-get install osm2pgsql
2. Load data
This repo contains the supplementary data to load and the queries that are issued to the database for each layer.
git clone https://github.com/mapzen/vector-datasource.git
WARNING: If you are standing up your own instance of the Tilezen stack (rather than doing development), it's best practice to checkout the latest tagged release rather than running off
master. At the time of this writing that is
v0.10.3, so you'd
git checkout v0.10.3 to be on the same code base as the production Mapzen Vector Tile service. Similarly, you'd need to pin yourself against the related project's versions, e.g.:
Requires: tileserver v0.6.1 and tilequeue v0.9.0 and TileStache v0.10.1 mentioned in the release notes in the sections below.
Set up database
First, create the database. We use the database name 'osm' here, but you can use any, e.g. 'gis'.
createdb -E UTF-8 -T template0 osm psql -d osm -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'
Next, download the OpenStreetMap source data. You can use any PBF, but we use a Mapzen metro extract here to get started.
Load PBF data
osm2pgsql -s -C 1024 -S vector-datasource/osm2pgsql.style -j path/to/pbf -d osm -H localhost
You may also need to pass in other options, like -U or -W, to ensure that you connect to the database with a user that has the appropriate permissions. For more details, visit the osm2pgsql wiki page and the postgresql docs for creating a user. You may need to check your connection permissions too, which can be found in the pg_hba.conf file.
Note that if you import the planet, the process can take several days, and can consume over 1TB (2TB is preferred cause need more space to prepare the database) of disk space at the time of writing. The OSM planet gets bigger every week, so it might be necessary to do a few trial runs to find out what it takes today. Have a look at some performance tuning docs for recommendations.
Load additional data and update database
vector-datasource/data directory contains scripts to load additional data and update the database to match our expected schema.
- Additional land & water data comes from openstreetmapdata and is under the same ODbL as the primary OpenStreetMap data.
- Low-zoom data from naturalearthdata.com is under public domain.
# Go to data directory cd vector-datasource/data # Build the Makefiles that we'll use in the next steps python bootstrap.py # Download external data make -f Makefile-import-data # Import shapefiles into postgis ./import-shapefiles.sh | psql -d osm # Add indexes and any required database updates ./perform-sql-updates.sh -d osm # Clean up local shape files make -f Makefile-import-data clean
Note that you may have to pass in a username/password to these scripts for them to connect to the database. Anywhere
-d osm is specified, you may need to also pass in
-U <username> and perhaps set a password too. For example, if my username is "foo" and my password is "bar", here's what I would do:
export PGPASSWORD=bar ./import-shapefiles.sh | psql -d osm -U foo ./perform-sql-updates.sh -d osm -U foo
3. Serve vector tiles
# dev packages for building sudo apt-get install build-essential autoconf libtool pkg-config # dev packages for python and dependencies sudo apt-get install python-dev python-virtualenv libgeos-dev libpq-dev python-pip python-pil libmapnik2.2 libmapnik-dev mapnik-utils python-mapnik
git clone https://github.com/mapzen/tileserver.git
Prepare a virtualenv
# Create a virtualenv called 'env'. This can be named anything, and can be in the tileserver directory or anywhere on your system. virtualenv env source env/bin/activate
Install Python dependencies
Install dependencies for tileserver
pip install -U -r tileserver/requirements.txt (cd tileserver && python setup.py develop)
Install tilequeue in development mode
git clone email@example.com:tilezen/tilequeue.git (cd tilequeue && python setup.py develop)
Finally, there's some code in
vector-datasource as well, which needs to be installed.
(cd vector-datasource && python setup.py develop)
cd ../tileserver cp config.yaml.sample config.yaml # update configuration as necessary edit config.yaml
Load Who's on First neighbourhood data
Finally, neighbourhood data is required to be loaded from Who's on First.
wget https://s3.amazonaws.com/mapzen-tiles-assets/wof/dev/wof_neighbourhoods.pgdump pg_restore --clean -d osm -O wof_neighbourhoods.pgdump
This will load a snapshot of the neighbourhoods data.
You should periodically update the Who's On First neighbourhoods data by running the following:
wget https://raw.githubusercontent.com/mapzen/tilequeue/master/config.yaml.sample -O tilequeue-config.yaml wget https://raw.githubusercontent.com/mapzen/tilequeue/master/logging.conf.sample tilequeue wof-process-neighbourhoods --config tilequeue-config.yaml
python tileserver/__init__.py config.yaml
You're ready to help us improve the Tilezen project! Please read our CONTRIBUTING.md document to understand how to contribute code.
Need to confirm your configuration? A test suite is included which can be run against a tile server.
Sample test URLs
Keeping up to date with osm data
Generally speaking, tile service providers make the trade-off to prefer generating stale tiles over serving the request on demand more slowly. Mapzen also makes this trade-off.
A lot of factors go into choosing how to support a system that remains up to date. For example, existing infrastructure, tolerance for request latency and stale tiles, expected number of users, and cost can all play roles in coming up with a strategy for remaining current with OpenStreetMap changes.
If you are on a particular release and would like to migrate your database to a newer one, you'll want to run the appropriate migrations. Database migrations are required when the database queries & functions that select what map content should be included in tiles change.
Note that the migration for each release in between will need to be run individually. For example, if you are on v0.5.0 and would like to upgrade to v0.7.0, you'll want to run the v0.6.0 and v0.7.0 migrations (we don't provide "combo" migrations).
# in this example, we're on v0.5.0 - checkout the migration to v0.6.0 git checkout v0.6.0 bash data/migrations/run_migrations.sh -d osm # now our database reflects v0.6.0 - checkout the migration to v0.7.0 git checkout v0.7.0 bash data/migrations/run_migrations.sh -d osm # now our database reflects v0.7.0