Maps no longer updating on new vector tile server, out-of-date with quests

[matkoniecz]

Yes, #835 is back and #747 continues to cause problems.

Firstly, thanks to @Akasch for his great work in setting up a new vector tile server for rescue StreetComplete from the demise of Mapzen.

I am opening this issue to mention that as the server is not updating at the moment (and I'm sure it is probably not trivial to do so easily or cheaply), and therefore the map is no longer up-to-date with the OSM data (and therefore StreetComplete quests).

Is it possible to restart updates or run cyclic updates (once a month? once 3 months?) or switch to an alternative background map to StreetComplete that pulls direct from osm.org rendered tiles?

Currently it is not fatal problem in areas that are not significantly changed but it will steadily get worse if nothing is changed.

[matkoniecz]

I tested it by reinstalling app (what purges data) and downloading area where I changed well-visible data over 10 days ago ( https://www.openstreetmap.org/way/134071005#map=17/50.07755/19.93830 in that case ).

From what I know data was supposed to be updated continuously, what worked for some time and stopped.

I am not sure is it possible that data update will return in any form (even limited one like reload every 3 months).

[rugk]

Hmm, it all looks as if we maybe need a better powerful server there.
But @westnordost still rejects any donations. Or another method here. Or you can e.g. apply for the Thomas-Krenn-Award, which not only brings publicity, but they also donate hardware…

[westnordost]

A server is not a problem, terrestris runs one and the person maintaining it said it would be ok if SC used it. But the format is a little different, i.e. the stylesheet would need to be changed.

[westnordost]

Is this still an issue? I think the server just had a hickup, maybe?

[dbdean]

Absolutely, the maps in my area haven't fully updated in at least 4 months.

[westnordost]

Hm, damn. The admin of the server also told me that he is quite busy right now (with other things), so no chance one can expect this will be solved anytime soon

…

On May 28, 2018 12:06:31 AM GMT+02:00, David Dean ***@***.***> wrote: Absolutely, the maps in my area haven't fully updated in at least 4 months.

[matkoniecz]

Since I created this issue (and weeks before) I have seen no updates and plenty of cases of outdated map.

[rugk]

A server is not a problem, terrestris runs one and the person maintaining it said it would be ok if SC used it. But the format is a little different, i.e. the stylesheet would need to be changed.

Well…, but if you run your own server, you could get everything in the format you like.

[davidak]

And when multiple trusted members would have access to the server, problems can be solved faster, even if one don't have the time. Documentation is also important, in best case directly in configuration management. Minimizing the bus-factor...

[Akasch]

I have deployed a test, now Germany should be a more recent (I think 23.4. catching up, and the old tiles will be cached between 7 and 14 Days), all other Data is more old.

If this test is working it would be possible to split the Server to multiple small ones. As small Server are a lot cheaper for combined capacity, and it is easy possible to use unused capacity on private virtual servers this could be a model for a community driven Tile-server. I know that at the moment there will be a Problem with the tiles on the Border, I will fix it in the next days.

I think that we should put the Discussion about the Tileserver into its own repo/issue tracker. I would put a repo with configs etc up in the next day at the https://github.com/Map-Data Organisation.

[rugk]

Well, a "community-driven" sever would have privacy problems, so is rather now who runs the server I get tiles from.

[matkoniecz]

would have privacy problems

What would be worse than it is now? Location of given user is identifiable anyway by analysis of changes made.

[Akasch]

At the Moment only the fronted Cache Server is getting the IP address from the User, the Server Producing the Tiles is only seeing the IP of the fronted web Servers, and at the moment the User Agent which is always "okhttp/3.5.0" for StreetComplete. I think that it is not possible to correlate users to requests.

The Structure at the moment is like this:

Cacheserver01 -----               |------------------> Tileserver Germany
                   |-----> Master Server
Cacheserver02 -----               |------------------> Tileserver World

The Master Server is routing the requests to the correct Tilesserver, and is serving the prerendered Tiles. For a better availability the master server could be duplicated.

[matkoniecz]

In that case there is almost no reason to consider privacy as an issue.

[rugk]

Ah yes, if users could just host the backend servers yeah. This would be useful.

To answer the previous question: users may/do not always answer quests, but when they have the app open tiles are always downloaded.

[westnordost]

Tiles seem to not load anymore, @Akasch

[Akasch]

@westnordost deployed a workaround, should work again, will look into the causes later

[dbdean]

Any movement on getting worldwide up-to-date maps happening again?

[cstackpole]

I was just in an area that I haven't been to in a long time. The tiles were all downloaded. Yet I had house quests without outlines. I too request a up-to-date map. Thank you for everyone helping out with this!

[francois2metz]

I would be happy to host the France tile server (or more depending of the load). The proposition to route each traffic country to a specific instance is nice IMHO.

But I have no idea how to setup a tile server and update it. Anyone to give me a link to some doc?

[pkoby]

I've moved to a new area and also have no information on my maps, despite having added a lot of the buildings to OSM a few months back. This is not really workable, as others have said, without the context for which building you're updating.

I understand the issues with servers and whatnot, but perhaps we could get an option to use offline maps? I use c:geo for geocaching, and it has this option. You can download maps from http://download.mapsforge.org/maps/, then point the app to use them instead of, say, Google maps or Mapnik via wifi or data.

The size of the maps is not trivial, but I'd like the ability to update the basemap as necessary by downloading the mapsforge option much more frequently (about every month they update).

EDIT: alternatively, Maps.me has a download page (http://direct.mapswithme.com/regular/daily/), but I'm not sure what the license would be on that.

[matkoniecz]

offline maps are discussed in #122 (tl;dr: "This is not possible yet with the map rendering library I use").

[westnordost]

Actually, it would be when implemented manually. A nested nested for-loop that downloads all tiles from zoom level A to B within a bounding box C,D,E,F and stores these tiles in a local SQLite database. Tangram-es can then be pointed at that database for offline usage. However, that is not an effort I am prepared to make, because I do not want to invest time and effort into the dealing with the map library.

(Needless to say, this would put quite a strain on the tileserver. I have the impression that even now, the map-data tileserver we use does not cope with the traffic)

[rugk]

I have the impression that even now, the map-data tileserver we use does not cope with the traffic

Only @Akasch can say for sure.

[ENT8R]

Is there something which speaks against using Mapbox Vector Tiles? I know that one has to adapt the stylesheet, but that should be no big problem... But at least those tiles are always up-to-date...

[matkoniecz]

The MapBox tileserver is only allowed to be used through the MapBox SDK. I initially decided against MapBox because they have some telemetry built-in and their terms and conditions require to have this by default on (only letting the user switch it off in the settings).

#747 (comment)

[arbie65]

Is there anything we as non technical users of streetcomplete can do to help get this ticket closed? It's a great tool but when features are in sync with OSM as it looks daily instead of a snapshot of months ago

[westnordost]

Not really. A more reliable and current map would require using a commercial vector tile service like Thunderforest, Omniscale, OpenMapTiles or what other providers there are. One could tentatively ask them if they would be willing to sponsor this app by providing the service for free to StreetComplete but this is probably something that I should do myself.

[mattwelke]

@westnordost Have you considered hosting a tile server in the cloud and reaching out to an organization like CloudFront to see if they'd sponsor free CDN service wrapped around your tile server to minimize load on it? DigitalOcean in particular often sponsors nonprofit organizations (https://www.digitalocean.com/community/questions/discounts-or-support-for-nonprofit-organisation), which could provide the origin server.

[westnordost]

No, because I do not want to concern myself with administrating a live tile server, rather I spend my time on developing the app.

[iandees]

FYI Nextzen just finished updating the data behind our tiles. I think the date on them is December 5th or so.

[westnordost]

Cool! So, how long did it take? Is this now a regular update or did you manually trigger it?

[iandees]

how long did it take?

It takes approximately a week to rebuild the tiles.

Is this now a regular update or did you manually trigger it?

We still have to manually trigger it. We're working on funding right now, but we might be able to get to quarterly or monthly update cycles.

[Susurrus]

What's the process for requesting or helping with updating these tiles again? I've been making substantial edits in my area, and the quests are updating, but the map tiles are pretty out of date at this point.

[Akasch]

At the moment unfortunately I have not so much time. But I now have a collection of scripts to setup a tileserver serving a limited region. Together with some server components it is now possible to distribute the tileserver. I will cleanup the scripts in the next days. Maybe put some components into a doker container for easier deployment. A server will need something like 50GB harddisk, 2-4 CPU cores, 4 GB RAM, I think. With CPU and RAM I am not so sure at the moment. Required software is Python-virtualenv (Python2), bash and postgresql with hstor and gis extensions. The 50 GB is for the bigger regions, there are multible smaller ones, the computing recurces are only needed at import time and for very short times while generating some tiles. So it should be possible to have two or more regions on a server if the harddisk is big enough. At the moment I am testing it with the region Hamburg in Germany. If there is someone who want to try hosting a region, write me, and we can try out the progress.

[Julientroploin]

New user of StreetComplete, I think I experience the bug described here, so I can not contribute to OSM with your app... An image of what I get while zooming : https://drive.google.com/file/d/1rArOT_YmfNMIU23lAiQPcCV5iHcZtBUo/view?usp=sharing
Reading a part of the discussion (and not being an english native-speacker), I think I understood that your stuck with a server problem.

I can not help but I wanted to thank all of You for your efforts.

[Akasch]

@Julientroploin can you give me the location of the error, then I can try to debug the error.

[Julientroploin]

Thank you for your fast answer, this affect a lots of tiles near my position (La rivière-de-corps, FRANCE 48° 17′ 19″ N, 4° 01′ 14″ E )
Hope you can help to debug this!

[Akasch]

Ok i can see the error, but I will need 2 or 3 days to fix it.

[Julientroploin]

Thank you @Akasch , I'll check this in a few days.

[Julientroploin]

@Akasch, It work now... I suppose you fix the problem, that's great ! Right in time to contribute under a fantastic spring sun !

[Akasch]

@Julientroploin yes, it was faster than i thought.

[NonnEmilia]

@Akasch also in my zone in Italy we have an old map not showing some recent (1 or 2 months old) roads. Could you kindly take a look?
Coordinates: Lat 44.76617 Lon 10.66540

[Akasch]

The scripts to host a regional part of the tileserver are finally mostly usable. So if you want your region to be updated you can contribute to it. The scripts can be found at: https://github.com/Map-Data/regiontileserver . You will need a Server with ~60 GB of free space, best on an SSD as the import will be faster (Hours instant of Days). Before starting the import please write me a short email so that we do not have unnecessary duplicates.

If you have questions ask me. If you do not want do do the administration yourself but sponsor a region I can do the administration on a provided server/virtual machine, contact me to discuss details.

A visualization of the hosted regions is available at https://map-data.de/#map=3/0/0&style=cinnabar&show_server . The red regions are served by http://nextzen.org with older data. The green ones are currently served by us, with data from the last few weeks. The yellow ones are normally served by us but due to maintenance currently not available so they fall back to nextzen.

[westnordost]

We have another ticket open for this, so I'll close this old one

[Kezxo]

We have another ticket open for this, so I'll close this old one

What ticket is that? I can't find any open ones linked here.

EDIT: is it #1700 ? It probably is.

[rugk]

Duplicate of #1700