Add /refresh to update sources and some configurations

[mesudBe-Orbit]

Try to fix #288

• Inject Arc<RwLock> to server, T: state.tiles, state.sprites, state.fonts, catalog, config
• Add a /refresh to endpoint(read and merge config and flush the cache)
• Add test to tesh.sh
• Update doc

[nyurik]

looks like a good start, thanks! I will have to think of how to test this best - perhaps add a new shell script test-watch.sh that is similar to existing test.sh, except that it will:

• start martin with --auto-bounds skip (so that it starts faster)
• wait for it to start
• get a catalog into catalog1.json
• use psql to create a new table and a function, and possibly make some other modifications like deleting something else
• use curl to POST a refresh command to some Martin's endpoint that triggers the refresh - I guess this request should not return until refresh is done? Make sure to have some global "refresh lock" to avoid multiple simultaneous refresh requests?
• get another catalog saving it to catalog2.txt

The existing bless/diff setup will check that the catalog results are as expected

[sharkAndshark]

Some ways to push this forward?

[sharkAndshark]

I'm trying to keep going on this.

[nyurik]

There is a well known dashmap crate that essentially functions as a concurrent hashmap (i.e. replaces RwLock<HashMap<K, V>>). We may want to use that as a simpler alternative to store sources / fonts / sprites. Or it might be a bad idea - only time will tell.

[sharkAndshark]

Weired. That shouldn't fail. CI / Build and test docker images

Testing points3857_srid_0_0_0.pbf from http://localhost:3111/points3857/0/0/0
***** Test server response for PMTiles source *****
Testing pmt.json from http://localhost:3111/stamen_toner__raster_CC-BY-ODbL_z3
curl: (22) The requested URL returned error: 404

The source stamen_toner__raster_CC-BY-ODbL_z3 has been configured actually.

[2024-06-05T02:55:58Z WARN  martin::utils::id_resolver] Source `stamen_toner__raster_CC-BY+ODbL_z3` (/tests/fixtures/pmtiles/stamen_toner__raster_CC-BY+ODbL_z3.pmtiles) was renamed to `stamen_toner__raster_CC-BY-ODbL_z3`. Source IDs must be unique, cannot be reserved, and must contain alpha-numeric characters or `._-`
[2024-06-05T02:55:58Z INFO  martin::file_config] Configured source stamen_toner__raster_CC-BY-ODbL_z3 from /tests/fixtures/pmtiles/stamen_toner__raster_CC-BY+ODbL_z3.pmtiles

[sharkAndshark]

There must be a lot to clean up.

[sharkAndshark]

Hi @nyurik Any guidance or review on this PR? I think the first version has been finished

[nyurik]

Thx @sharkAndshark! I will need a bit of time to review, a bit swamped at the moment, but I don't want to miss anything important for such a critical PR!

[nyurik]

First - awesome work! Some serious time and effort were done on this one. Thx!

Now to the "fun" part:

• We must not have multiple write() calls on multiple objects together like you have in the refresh because it is extremely slow - you get a write lock on one object, and try to get a write lock on another object -- while holding on to the first one. Getting a write lock means you have to wait for all the readers to finish. But you already hold on to the first write lock - so no other readers can use the first object while you are waiting. Really bad for performance, and could result in a deadlock in some cases.
• It seems Data<RwLock<T>> pattern is not good - creates too many complexities with locking.
• Instead, we should probably go with Data<Arc<DashMap<String, Box<dyn Source>>>> (and similar for sprites and fonts). This way you never need to deal with read/write locks (handled by DashMap), each source is independent, and the DashMap instance is always the same.
• On refresh, we will need to do these steps:
  • discover all new sources, e.g. new PG tables
  • for each discovered source, insert it into dashmap under its name
  • TBD: I don't know if it makes sense to compare old source with the new source, and only replace if they are different. There might be caching reasons not to replace if the source has not changed.
  • TBD: We need to delete sources that no longer exist. Not easy -- we may need to add a source group ID to each source -- e.g. pg-1 and pg-2 for the first and second postgres connections. When refreshing pg-1, we can iterate over all sources, and remove those that have pg-1 and were not auto-discovered.
• To keep things cleaner, we could create a new TileSources struct, and maybe implement Deref on it.

pub struct TileSources(Arc<DashMap<String, Box<dyn Source>>>);

• we should not re-create cache on refresh. Ideally we should only purge objects related to the specific source if that source is replaced. At the moment, I don't think we ever cache combined tiles, so that's not an issue.

In light of all the above, I propose our next steps would be to break this PR into several parts - lets try to do just one simple type of "refreshable" first -- like fonts or sprites. It makes no sense to do all of them at once until we are certain how we can proceed.

P.S. I made a few minor cleanup commits directly to your branch, and merged with the latest main.

[sharkAndshark]

Thx! Let's do it step by step.

[nyurik]

@sharkAndshark can i help with anything on this one - I know it is a bit disappointing that we can't merge it like this, but it will be a massive performance hit if we do it this way.

[sharkAndshark]

Thx! Do anything you like. 😉I did some try in another branch 2 weeks ago(as a fresh start might be easier), but I have no too much time recently. @nyurik