Create a data caching server to speed up data loading

[pietervdvn]

One of the NL-Net tasks is to speed up data loading

Introduction

Currently, MapComplete loads all data from Overpass (or from the OSM-AP if zoomed in). For
some themes, it would be nice to see globally how much data is available and to be able to load
the data quickly. While Overpass is an amazing service and allows extremely flexible queries, it
often takes a long time to load the data – especially on a global level.
A service like MapComplete doesn’t need this flexibility: once a thematic visualization is created, it
is well-known what features should be loaded. A service could then cache all these features and
keep these layers up-to-date with the minutely diffs. This also ties in with the internal handling of
data: within MapComplete, the ‘FeaturePipeline’ is a complicated piece of code which merges the
data from different sources (overpass, fallback-overpass, cache, external sources, the actual
OSM-API). It contains several bugs, and issues, for example: it can’t handle deleted points nor can
and emits many duplicate features in some cases.

Tasks

• Find and adapt a vector tile generating server, with a data layer for every mapcomplete-
  layer
• Refactor the feature pipeline and feature sources to remove unneeded code
• Fix caching behaviour and behaviour of deleted items

As this issue is mostly for project management, no discussions are allowed here. Please use #581 for technical discussions

[thibaultmol]

@pietervdvn this is solved now I think?

[pietervdvn]

Indeed!