[WIP] Updater

[SlowMo24]

Write update-package that fulfils framework set out here: https://confluence.gistools.geog.uni-heidelberg.de/display/oshdb/Update+process+framework

Missing implementations are:

Updater Package

• implement different backends (ignite, h2 etc)
• insert new tags into tagtranslator-db BEFORE creating new entity
  • update TagTranslator cache if new tag or role is inserted to keytables or handle dublicate insertion exception
• account for back-references of OSHEntites (e.g. ways for nodes for changed ways and changed nodes)
  • account for back-references of backreferences of OSHEntites (e.g. relations depending on ways depending on changed nodes)
  • Relations are excluded for now, but might get important in a later update? @rtroilo
• implement flushing of updates to real database
• implement Kafka promoter (do we actually want this, its not hard to do but also a feature with no use-case so far)
• collect OSHEntites before processing to avoid multiple calls to same method
  • test if this actually is an improvement
• collect OSHEntites into insertCells before flushing to prevent multiple overwriting of the same Cell
• resolve comments in code
• assure concurrency (most of all, check if locking in flusher is safe)
• optimise Code (very important last step!)
• analyse and clean dependencies
• extensive testing!
• codeStyle, staticCodeAnalyses etc (resrouce handling)
• documentation
• correct errors introduced by 2ab2f88
  • check if functionality was preserved
• there might be a mixup of CellId.getId() and CellId.getLevelId() in the Flusher and possibly elswhere! Should be investigated!

OSHDB-API

• write class to append entities from update-db to stream when querying over OSHDB-API (I need help to point me to the right place where this would be implemented @tyrasd ?)
  • implement ScanQuery and LocalPeek OSHDB-API-Ignite Backends
  • move streaming of updates up do MapReducer-Class. It is only copy code now.
  • group 100 or 1'000 entities together in one 0,0 GridCell in OSHDB-API to avoid overhead of CellCreation (and calling of celliterator methods)
• improve testData for updates to speed them up (Area to big atm)

ETL-Backend

• implement ETLFileReaderClass (help needed from @rtroilo )

Dependencies

• check if ignite-geospatial is actually necessary as a dependency or only needs to be available at cluster startup on the server side. If so, remove artifact from artifactory and remove dependency.
• check if h2gis is necessary as a dependency

[SlowMo24]

😢😢😢