Dockerfile for osm2pgsql #1428

hypervtechnics · 2021-03-13T14:34:16Z

hypervtechnics
Mar 13, 2021

As part of the MapDiscover project we are planning to use osm2pgsql to create a POI database. The Lua integration is just perfect for this use case. Thanks for this one! MapDiscover consists of multiple services, each represented as a container image.

So we tried to build a container image using Docker + Dockerfile and the build instructions of the README.md. As mentioned in #1427 the image size is very large (unoptimized ~1,4 GB with debian-slim-buster). I tried to use multi-stage builds and so on, but only could get to 700 MB by manually (using try and error) removing several dependencies (still not sure if everything works):
There are also no official docs which libraries are needed to run the binary release once you built it.

Today we got the first working docker image with several additions:

Continuous replication support
LuaJIT support
Initial parsing of osm input
Usage of environment variables

I am somehow used to Golang where you built an application and just put it in the container. Of course this doesn't work with C++ - I know. But I think having 700+ MB of dependencies for running an application seems just too much.

EDIT: I understand that providing a Dockerfile as part of the project requires time to maintain. Although I think clear build instructions and information about dependencies should be mentioned somewhere.

joto · 2021-03-13T14:37:58Z

joto
Mar 13, 2021
Maintainer

You can use the ldd command with any binary to figure out what libraries it needs.

1 reply

hypervtechnics Mar 13, 2021
Author

Thanks for this information! Didn't know about that.

ImreSamu · 2021-03-13T15:06:26Z

ImreSamu
Mar 13, 2021

the image size is very large (unoptimized ~1,4 GB with debian-slim-buster).

my2c:

IF the latest osm2pgsql release ( 1.4.1 ) is ok ..
- THEN --> my proposal the debian:bullseye packaged version!
I am using the bullseye everywhere ( as a base Docker image ) ..
- it is ~stable and contains the latest releases from osmium / osm2pgsql / ...

simple example:

Now:

2021-03-13 14:53:02 osm2pgsql version 1.4.1
size: 123MB

$ docker images | grep osm2pgsql
osm2pgsql                              bullseye          3f3d972b5a11   37 seconds ago   123MB

Dockerfile

ARG BASE_IMAGE=debian
ARG BASE_TAG=bullseye-slim
FROM ${BASE_IMAGE}:${BASE_TAG}

ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_US.UTF-8

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    osm2pgsql \
  && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/*

RUN osm2pgsql --help

build: docker build -t osm2pgsql:bullseye .

1 reply

hypervtechnics Mar 13, 2021
Author

Thank you for your contribution! This works quite good. Didn't know they already released bullseye to Docker Hub.
Unfortunately I need to compile osm2pgsql from source as I need to have LuaJIT enabled. We rely heavily on the flex output and the logic in the Lua code will grow in time.

Maybe there is a JIT enabled version? Or maybe make could generate some kind of deb file which would reference the dependencies in the repositories and therefore solve the dependency problem.

remyleone · 2021-03-15T09:52:17Z

remyleone
Mar 15, 2021

I think we could do this work in 2 steps:

As a first step, we should have the option in the build tool of osm2pgsql (cmake) to build the osm2pgsql binary from the sources statically. This would allow the creation of a single binary with no dependencies on external libraries. This is very similar to what golang does natively. This would be completely integrated into the already existing CMake build tool. So no new scripts there, only options added to an already existing script.
As a second step, using a multi-stage build to build from the sources (including with Lua support) and then placing the created binary in a container. This new Dockerfile would be a copy and paste from the build instructions that are already given and maintained in the README.md

This feature could really help develop a container-based workflow for ingesting data into postgis.
Before investing more time doing those contributions, I would like to know if the maintainers of the osm2pgsql would merge contributions that offer these two features? @lonvia @joto

3 replies

lonvia Mar 15, 2021
Maintainer

Why would you need docker if you have a static binary? You can just copy the binary around and use it. That said, osm2pgsql can't be build statically because the Postgresql libraries aren't made to be used statically. You'd have to start with building a special version of the Postgresql libraries and that is way out of scope of this repository.

hypervtechnics Mar 15, 2021
Author

Sounds like a very good idea! Unfortunately I am not that familiar with C++ development, so I can not contribute with this other than copying Makefiles from somewhere else :/

remyleone Mar 15, 2021

If you have a container-based workflow for describing your work, you can simply specify a container such as openstreetmap/osm2pgsql and have a container working natively without managing the binary by itself.

ImreSamu · 2021-03-15T11:21:05Z

ImreSamu
Mar 15, 2021

@hypervtechnics :

V2. "extreme minimalized version" . ~ 47.5MB

latest osm2pgsql master
alpine edge - with the latest compilers ( not stable! )
luajit 2.2.0-r3 ???
multi-layer + ldd config tricks
march=native -o3 lto ... ( cargo cult optimization ... just delete )

not tested .. ( imho: need a full test for the native optimisation )
for me : ~ 47.5MB

# experimental;  save as a 'Dockerfile'
# build:       docker build -t osm2pgsql_luajit . 
FROM alpine:edge as alpine_build
RUN apk --update-cache add cmake make git  g++ boost-dev expat-dev bzip2-dev zlib-dev libpq proj-dev postgresql-dev  luajit #  lua5.3-dev
# build osm2pgsql latest master
RUN git clone --depth 1 --branch master https://github.com/openstreetmap/osm2pgsql.git
WORKDIR /osm2pgsql/build
RUN mkdir -p /osm2pgsql/build

############################
# <CARGO CULT ON>
# .. cpu native + lto optimization .. only for the local testing .. not portable! 
# .. not recomended for normal users.
ARG COMPILE_FLAGS=" -march=native -O3 -flto -fno-semantic-interposition -ffat-lto-objects "
ENV CFLAGS=${COMPILE_FLAGS}
ENV CXXFLAGS=${COMPILE_FLAGS}
ENV OBJCFLAGS=${COMPILE_FLAGS}
ENV FFLAGS=${COMPILE_FLAGS}
ENV LDFLAGS="-flto=auto"
# <CARGO CULT OFF>
#######################

RUN cmake .. -D WITH_LUA=OFF -D WITH_LUAJIT=ON
RUN make -j$(nproc)
RUN make install
RUN /usr/local/bin/osm2pgsql --help
RUN ldd /usr/local/bin/osm2pgsql
# "ldd tricks" - credit:  https://gist.github.com/bcardiff/85ae47e66ff0df35a78697508fcb49af#gistcomment-2078660
RUN ldd /usr/local/bin/osm2pgsql | tr -s '[:blank:]' '\n' | grep '^/' | \
    xargs -I % sh -c 'mkdir -p $(dirname deps%); cp % deps%;'


FROM alpine:edge
# copy deps .. generated by "ldd tricks"
COPY --from=alpine_build /osm2pgsql/build/deps /
# copy osm2pgsql
COPY --from=alpine_build /usr/local/bin/osm2pgsql                 /usr/local/bin/osm2pgsql
COPY --from=alpine_build /usr/local/share/osm2pgsql/default.style /usr/local/share/osm2pgsql/default.style
COPY --from=alpine_build /usr/local/share/osm2pgsql/empty.style   /usr/local/share/osm2pgsql/empty.style
COPY --from=alpine_build /usr/local/share/man/man1/osm2pgsql.1    /usr/local/share/man/man1/osm2pgsql.1
# smoke test
RUN  /usr/local/bin/osm2pgsql --help
RUN  ldd /usr/local/bin/osm2pgsql
CMD ["/usr/local/bin/osm2pgsql"]

1 reply

1ed Jun 27, 2021

These can be removed as they exists in the base layer as well and the size got down to ~41MB on my machine.

RUN rm -f /osm2pgsql/build/deps/lib/libcrypto.so.1.1 /osm2pgsql/build/deps/lib/ld-musl-x86_64.so.1 /osm2pgsql/build/deps/lib/libssl.so.1.1 /osm2pgsql/build/deps/lib/libz.so.1

nyurik · 2021-10-28T15:57:18Z

nyurik
Oct 28, 2021

I think it would be great if dockerfile is maintained by osm2pgqsl community or the shared docker library. I see several benefits:

a user on any OS can quickly run any needed osm2pgsql version without dealing with complex build steps or relying on (often dated) packages.
any casual osm2pgsql contributor can quickly test their changes if they can run same build & test steps as ran by the CI, without the need to set up all build requirements -- as that's a significant investment especially for casual development
new releases can publish to the docker hub automatically by github actions the moment the release is made.
latest main branch rev (and other significant branches) can also autopublish for testing and experimentation purposes.
anyone who wants to build their own custom images can quickly see the steps needed to get code to compile.

Note that this is the same discussion as happened at PostGIS community, and despite some initial resistance, we now have a well maintained variety of postgis images for multiple platforms.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dockerfile for osm2pgsql #1428

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 6 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Dockerfile for osm2pgsql #1428

hypervtechnics Mar 13, 2021

Replies: 5 comments · 6 replies

joto Mar 13, 2021 Maintainer

hypervtechnics Mar 13, 2021 Author

ImreSamu Mar 13, 2021

Dockerfile

hypervtechnics Mar 13, 2021 Author

remyleone Mar 15, 2021

lonvia Mar 15, 2021 Maintainer

hypervtechnics Mar 15, 2021 Author

remyleone Mar 15, 2021

ImreSamu Mar 15, 2021

V2. "extreme minimalized version" . ~ 47.5MB

1ed Jun 27, 2021

nyurik Oct 28, 2021

hypervtechnics
Mar 13, 2021

Replies: 5 comments 6 replies

joto
Mar 13, 2021
Maintainer

hypervtechnics Mar 13, 2021
Author

ImreSamu
Mar 13, 2021

hypervtechnics Mar 13, 2021
Author

remyleone
Mar 15, 2021

lonvia Mar 15, 2021
Maintainer

hypervtechnics Mar 15, 2021
Author

ImreSamu
Mar 15, 2021

nyurik
Oct 28, 2021