-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Planet state file #6
Comments
Hi, For the life of me if I can figure this out. As a background, to understand my approach, we're building a private OSM server and a tiles server, part of a bigger app. I can't manage to generate the correct state.txt. Using osmium fileinfo on the generated PBF shows the latest change's timestamp, but no sequenceNumber. To summarise, when a new client joins:
I'd appreciate some help, thanks :) |
The There are tools to synchronise a planet dump with a chosen replication stream, for example pyosmium's up-to-date tool. This works by looking at the timestamp of the planet file, rewinding a bit and replaying the diffs covering that period. The general reason why these streams are all independent is that it previously wasn't easy to identify a linear point in time in Postgres, hence all the stuff in Osmosis' state file about In summary; Hope that helps! |
it does, thanks for the explanation. Edit: technically, I'd rather reset the sequence numbers every time |
Hi, I wanted to contribute to your project so I'm pasting our dockerfiles here. Maybe you guys need it. I removed line, might hiccup at permissions on the volume but I don't think so. Dockerfile: FROM debian:buster-slim
ARG PLANET_DUMP_URL=https://github.com/zerebubuth/planet-dump-ng/archive/v1.2.0.tar.gz
RUN set -eu; \
apt-get update; \
apt-get install -y --no-install-recommends \
build-essential \
autoconf \
automake \
ca-certificates \
curl \
libboost-date-time-dev \
libboost-dev \
libboost-filesystem-dev \
libboost-iostreams-dev \
libboost-program-options-dev \
libboost-thread-dev \
libosmpbf-dev \
libprotobuf-dev \
libxml2-dev \
osmpbf-bin \
pkg-config \
postgresql-client-11; \
useradd -u 999 -r planetdump; \
mkdir /opt/build; \
curl -sL $PLANET_DUMP_URL | tar xz -C /opt/build --strip-components=1; \
cd /opt/build; \
./autogen.sh; \
./configure; \
make -j $(nproc); \
make install; \
cd /; \
rm -rf /opt/build; \
mkdir /dumps; \
chown planetdump:planetdump /dumps
COPY entrypoint /usr/local/bin/entrypoint
VOLUME /dumps
USER planetdump
WORKDIR /dumps
ENTRYPOINT ["/usr/local/bin/entrypoint"]
CMD ["bash"] entrypoint (chmod +x) #!/bin/sh
set -eu
PBF_FILE=${PBF_FILE:-latest.pbf}
case "$1" in
dump)
cd /dumps
rm -rf users changeset* node* way* relation*
echo "dumping OSM db"
DUMP_FILE=$(mktemp)
pg_dump -F custom > $DUMP_FILE
echo "creating PBF"
planet-dump-ng -f $DUMP_FILE -p "$PBF_FILE"
rm -rf users changeset* node* way* relation*
;;
*) exec "$@";;
esac Edit: added missing file name, removed osmium. |
It's a pain to figure out which replication
state.txt
file corresponds to a given planet and, now that all the current & history for both XML and PBF correspond to the same state, it would make it easier if the dump process or script would figure out the state from which replication could continue.The dump already tracks the last timestamp in the file, and this can be used to find a state file. But there might be in-progress transactions at that point, so it will be necessary to track backwards in the state files until before all those transactions start.
The text was updated successfully, but these errors were encountered: