OpenStreetMap data to PostgreSQL converter
C++ Python CMake Other
Clone or download
lonvia Merge pull request #863 from lonvia/update-libosmium
update libosmium to 2.14.2 and protozero to 1.6.3
Latest commit 93b73e5 Aug 11, 2018
Permalink
Failed to load latest commit information.
cmake Command line option "-V" to include library versions Apr 22, 2018
contrib update libosmium to 2.14.2 and protozero to 1.6.3 Aug 10, 2018
docs Use HTTPS for planet links in man page Jan 10, 2018
tests add test for parsing with extra arguments Aug 5, 2018
.clang-format persistent store: catch invalid location exception Jan 7, 2017
.gitignore Clean up gitignore, now that all build files are in a build dir Oct 26, 2015
.travis.yml LuaJIT Travis CI Apr 14, 2018
AUTHORS Hoffmann has two n's Jul 7, 2015
CMakeLists.txt prepare for 0.96.0 release May 2, 2018
CONTRIBUTING.md document dependency on psycopg for testing Mar 2, 2018
COPYING * improved configure setup (including automake and libtool) Nov 6, 2010
ChangeLog Revert "Remove the old SVN Changelog" Oct 3, 2013
README Trim down README and move more detailed information into docs directory Sep 4, 2014
README.md prepare for 0.96.0 release May 2, 2018
appveyor.yml Fix an issue with the platform env variable Feb 27, 2018
default.style Small gramatic change Aug 9, 2016
empty.style remove whitespace before EOL Nov 2, 2015
expire-tiles.cpp uint32_t for zoom level in options_t, zoom level 0 for "no tile expiry" Apr 21, 2017
expire-tiles.hpp fix signed-to-unsigned warnings, use maxzoom=0 for "no tile expiry" Apr 21, 2017
geometry-processor.cpp Merge pull request #721 from lonvia/relation-member-filtering Aug 31, 2017
geometry-processor.hpp Merge pull request #721 from lonvia/relation-member-filtering Aug 31, 2017
id-tracker.cpp remove shared_ptr for id_trackers Mar 30, 2016
id-tracker.hpp remove shared_ptr for id_trackers Mar 30, 2016
install-postgis-osm-db.sh remove whitespace before EOL Nov 2, 2015
install-postgis-osm-user.sh remove whitespace before EOL Nov 2, 2015
middle-pgsql.cpp rework multi-threading during indexing Jan 13, 2018
middle-pgsql.hpp rework multi-threading during indexing Jan 13, 2018
middle-ram.cpp rework multi-threading during indexing Jan 13, 2018
middle-ram.hpp rework multi-threading during indexing Jan 13, 2018
middle.cpp Replace boost::shared_ptr with std::shared_ptr Jul 18, 2015
middle.hpp rework multi-threading during indexing Jan 13, 2018
multi.lua Correct spelling of superseded Dec 22, 2016
multi.style.json remove comments in json configuration files Nov 18, 2016
node-persistent-cache.cpp missing m_ prefixes break builds. Jan 12, 2018
node-persistent-cache.hpp Factor in clean_up() into the destructor. Jan 12, 2018
node-ram-cache.cpp clang-format of node-ram-cache Jan 7, 2017
node-ram-cache.hpp Remove unused usage of dirty-bit in ram node cache Jan 6, 2018
options.cpp Command line option "-V" to include library versions Apr 22, 2018
options.hpp uint32_t for zoom level in options_t, zoom level 0 for "no tile expiry" Apr 21, 2017
osm2pgsql.cpp do not handle extra attributes via parameter passthough Dec 11, 2016
osmdata.cpp always fully parse objects when extra attributes are requested Aug 5, 2018
osmdata.hpp always fully parse objects when extra attributes are requested Aug 5, 2018
osmium-builder.cpp catch exceptions by reference Jun 11, 2018
osmium-builder.hpp multi-output: reintroduce line spliting May 9, 2017
osmtypes.hpp don't pre-filter relation member tags Apr 22, 2017
output-gazetteer.cpp gazetteer: ignore name:suffix tag Apr 8, 2018
output-gazetteer.hpp rework multi-threading during indexing Jan 13, 2018
output-multi.cpp rework multi-threading during indexing Jan 13, 2018
output-multi.hpp rework multi-threading during indexing Jan 13, 2018
output-null.cpp rework multi-threading during indexing Jan 13, 2018
output-null.hpp rework multi-threading during indexing Jan 13, 2018
output-pgsql.cpp rework multi-threading during indexing Jan 13, 2018
output-pgsql.hpp rework multi-threading during indexing Jan 13, 2018
output.cpp replace OsmType with osmium::item_type Dec 11, 2016
output.hpp rework multi-threading during indexing Jan 13, 2018
parse-osmium.cpp ignore relations with more than 32k members Mar 13, 2017
parse-osmium.hpp improve constness of parsed osmium objects Feb 3, 2017
pgsql.cpp autodestruct PGresult Mar 25, 2017
pgsql.hpp autodestruct PGresult Mar 25, 2017
processor-line.cpp move builder from processors to output-multi Jul 25, 2017
processor-line.hpp move builder from processors to output-multi Jul 25, 2017
processor-point.cpp move builder from processors to output-multi Jul 25, 2017
processor-point.hpp move builder from processors to output-multi Jul 25, 2017
processor-polygon.cpp move builder from processors to output-multi Jul 25, 2017
processor-polygon.hpp move builder from processors to output-multi Jul 25, 2017
reprojection.cpp extend mercator projection to 89.99 latitude Oct 29, 2017
reprojection.hpp remove unused code Feb 4, 2017
sample.multi.json fix tab in sample multi config Dec 6, 2014
sprompt.cpp entirely avoid dynamic allocation for a single DWORD, put it on the s… Jun 2, 2016
sprompt.hpp Renamed C files to C++ in preparation for destaticification. May 12, 2014
style.lua adapt lua style to unfiltered relation members Apr 22, 2017
table.cpp Let postgres resolve non conflicting index names itself May 6, 2018
table.hpp fix counter initialisation in wkb_reader Aug 12, 2017
taginfo.cpp close style file on error Mar 10, 2017
taginfo.hpp allow more numeric datatypes for table columns in C tagtransform Nov 19, 2016
taginfo_impl.hpp output-multi: compute way_area only when there is a column with the name Feb 4, 2017
tagtransform-c.cpp fix uninitialised variable Feb 27, 2018
tagtransform-c.hpp don't pre-filter relation member tags Apr 22, 2017
tagtransform-lua.cpp Merge pull request #845 from mmd-osm/patch/luafunctionerrors Jun 11, 2018
tagtransform-lua.hpp don't pre-filter relation member tags Apr 22, 2017
tagtransform.cpp split tagtransform header into one file per class Mar 11, 2017
tagtransform.hpp don't pre-filter relation member tags Apr 22, 2017
util.cpp replace printf with boost::format Jan 16, 2015
util.hpp remove scale everywhere but in ramNode May 30, 2015
wildcmp.cpp rework wildcard code Feb 28, 2016
wildcmp.hpp rework wildcard code Feb 28, 2016
wkb.hpp use memcpy for potentially unaligned access to WKB Jan 6, 2018

README.md

osm2pgsql

osm2pgsql is a tool for loading OpenStreetMap data into a PostgreSQL / PostGIS database suitable for applications like rendering into a map, geocoding with Nominatim, or general analysis.

Features

  • Converts OSM files to a PostgreSQL DB
  • Conversion of tags to columns is configurable in the style file
  • Able to read .gz, .bz2, .pbf and .o5m files directly
  • Can apply diffs to keep the database up to date
  • Support the choice of output projection
  • Configurable table names
  • Gazetteer back-end for Nominatim
  • Support for hstore field type to store the complete set of tags in one database field if desired

Installing

Most Linux distributions include osm2pgsql. It is also available on macOS with Homebrew.

Unoffical builds for Windows are available from AppVeyor but you need to find the right build artifacts. For the latest release 0.96.0, you may download a 32bit version or 64bit version.

Building

The latest source code is available in the osm2pgsql git repository on GitHub and can be downloaded as follows:

$ git clone git://github.com/openstreetmap/osm2pgsql.git

Osm2pgsql uses the cross-platform CMake build system to configure and build itself and requires

Required libraries are

It also requires access to a database server running PostgreSQL 9.1+ and PostGIS 2.0+.

Make sure you have installed the development packages for the libraries mentioned in the requirements section and a C++ compiler which supports C++11. Both GCC 4.8 and Clang 3.4 meet this requirement.

First install the dependencies.

On a Debian or Ubuntu system, this can be done with:

sudo apt-get install make cmake g++ libboost-dev libboost-system-dev \
  libboost-filesystem-dev libexpat1-dev zlib1g-dev \
  libbz2-dev libpq-dev libproj-dev lua5.2 liblua5.2-dev

On a Fedora system, use

sudo dnf install cmake make gcc-c++ boost-devel expat-devel zlib-devel \
  bzip2-devel postgresql-devel proj-devel proj-epsg lua-devel

On RedHat / CentOS first run sudo yum install epel-release then install dependencies with:

sudo yum install cmake make gcc-c++ boost-devel expat-devel zlib-devel \
  bzip2-devel postgresql-devel proj-devel proj-epsg lua-devel

On a FreeBSD system, use

pkg install devel/cmake devel/boost-libs textproc/expat2 \
  databases/postgresql94-client graphics/proj lang/lua52

Once dependencies are installed, use CMake to build the Makefiles in a separate folder

mkdir build && cd build
cmake ..

If some installed dependencies are not found by CMake, more options may need to be set. Typically, setting CMAKE_PREFIX_PATH to a list of appropriate paths is sufficient.

When the Makefiles have been successfully built, compile with

make

The compiled files can be installed with

sudo make install

By default, the Release build with debug info is created and no tests are compiled. You can change that behavior by using additional options like following:

cmake .. -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=ON

Usage

Osm2pgsql has one program, the executable itself, which has 43 command line options.

Before loading into a database, the database must be created and the PostGIS and optional hstore extensions must be loaded. A full guide to PostgreSQL setup is beyond the scope of this readme, but with reasonably recent versions of PostgreSQL and PostGIS this can be done with

createdb gis
psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'

A basic invocation to load the data into the database gis for rendering would be

osm2pgsql --create --database gis data.osm.pbf

This will load the data from data.osm.pbf into the planet_osm_point, planet_osm_line, planet_osm_roads, and planet_osm_polygon tables.

When importing a large amount of data such as the complete planet, a typical command line would be

osm2pgsql -c -d gis --slim -C <cache size> \
  --flat-nodes <flat nodes> planet-latest.osm.pbf

where

  • <cache size> is about 75% of memory in MiB, to a maximum of about 30000. Additional RAM will not be used.
  • <flat nodes> is a location where a 36GiB+ file can be saved.

Many different data files (e.g., .pbf) can be found at planet.osm.org.

The databases from either of these commands can be used immediately by Mapnik for rendering maps with standard tools like renderd/mod_tile, TileMill, Nik4, among others. It can also be used for spatial analysis or shapefile exports.

Additional documentation is available on writing command lines.

Alternate backends

In addition to the standard pgsql backend designed for rendering there is also the gazetteer database for geocoding, principally with Nominatim, and the null backend for testing. For flexibility a new multi backend is also available which allows the configuration of custom PostgreSQL tables instead of those provided in the pgsql backend.

Contributing

We welcome contributions to osm2pgsql. If you would like to report an issue, please use the issue tracker on GitHub.

More information can be found in CONTRIBUTING.md.

General queries can be sent to the tile-serving@ or dev@ mailing lists.