Performance and refactor of Spark module

[timrobertson100]

Data growth has meant that the backfill jobs regularly take 9+ hours with 600 cores which is now a concern. Additionally, the code is convoluted and now difficult to maintain. Explorations in this proof of concept look promising and should be completed to provide a drop-in replacement of the module.

Some design goals:

• A drop-in replacement module (target Spark 2, but plan for Spark 3)
• Aim for a significant performance increase using the same or fewer resources
• Aim to remove scala code unless necessary
• Aim for simpler code to grok

[timrobertson100]

A first working skeleton for this is here
(1.2hrs for EPSG:3857)

[timrobertson100]

If building on M1, then installing the very old protoc can be done with these instructions and this patch

[timrobertson100]

Now deployed to DEV and UAT. Vector tile server full of:

WARN  [2023-10-24 14:41:14,811+0000] [qtp400103862-48969] org.gbif.maps.common.filter.VectorTileFilters: Unexpected non integer metadata entry null:121

Presumably year should be something other than null (0 perhaps?)

[timrobertson100]

Presumably year should be something other than null (0 perhaps?)

Confirmed as 0, and fixed in branch

[timrobertson100]

Now in production