Daily(?) updates for the vector tiles data (Berlin)

[tordans]

We have a map 🥳 . https://parkraum.osm-verkehrswende.org/project-vector-tiles/#16.5/52.513904/13.427681

One next step is, to get recurring updates going. Ideally hourly, maybe daily, maybe weekly…

---

Follow up to #10

[gislars]

just some thoughts:

• for daily updates
  The current scripts can be run as cron job. The berlin-extract from download.geofabrik.de will be downloaded and processed. Since the extracts are generated once a day, we would need an other solution to update the data.

• for updates less than a day
  For now, the scripts are run for a whole, fresh import and recreating all the tables. A continuously updated database can be setup with osm2pgsql. But for our post processing the way we update the data needs to be changed.
  For the future we might want to seed a cache with all the vector tiles. Currently we just use varnish and with every update we drop the varnish cache. So with an continuously updated database we would also like to update just a subset of vector tiles. It all depends on how fast the vector tiles can be (re)created. For this task we should have a look at https://github.com/onthegomap/planetiler

[tordans]

for updates less than a day

The PlanetTiler approach would be to not bother with partial updates and just re-run everything (but fast). I like the simplicity of that.

I understand now, we have two variables here

• the max time it takes for the data to process the given region
• the source of the data that we process, which also needs to be fresh (or running the script is pointless ;-))

My take:
Let's look at the simplest (daily?) solution for now as a next step…

[gislars]

Jede Nacht wird nun die DB automatisch aktualisiert, d.h. neues PBF von Geofabrik heruntergeladen und mit unseren Scripten dann verarbeitet. Den Cache-Control-Header habe ich auf 12 Stunden gesetzt, d.h. nach dieser Zeit sollte der serverseitige Cache (Varnish) neue Daten von pg_tileserve holen.

[joshinils]

Wie sieht es mit #23 aus?

[tordans]

Ich denke, wir können dieses Ticket schließen. Vielen Dank für die (fast) täglichen Updates, @gislars.

Für mögliche Folge-Themen wie Error-Handling wenn es keine neuen Daten bei der Geofabrik gibt (etc.) sollten wir dann bei Zeiten ein Folge-Ticket erstellen.