[Meta] Switch from Overpass queries to full OSM Data download + local processing

[RubenKelevra]

@westnordost wrote:

I'd rather have you open a new ticket because this is about something different now than how it started.

Alright, here we go:

StreetComplete currently runs one query per Quest-type to the Overpass-API to and generates pins from it.

Additionally, the map data is loaded from a map-server as a vector-format.

To conserve bandwidth and processing time, the amount of quests fetched is reduced. Additionally, the MapServer is sometimes overcrowded and out of date.

One possible solution is to download an area around the user from the regular API and process the data to generate the vector-format tiles from it.

Afterward, the same data can be processed to get the quests for the area.

This way the server load is very low and the processing time is mostly spent on the mobile device - this distributed approach does scale better with an ever-increasing number of quests, SC-users and mobile processing power.

Data consumption

The data consumption would be lower since we don't have to download the data multiple times for different zoom levels and different quests:

OSM download traffic for a typical area SC will download (in the city): 600 KByte.

Manually download all quests for an area (plus the default zoom level map after installation): 1,06 MB.

Filesize of the OSM data after decompression is 6.4 MB (disk space requirement)

Vector graphic tiles from OSM data

@westnordost wrote:

@westnordost do you think it's viable to build an osm to svg converter in the future, to avoid having to store the same data twice?

SVG? Ah I see what you mean. It would need to be an OSM to MBTiles converter. MBTiles in the same format as map-data.de/nextzen. @Akasch might know iif there is something usable for Java.

[westnordost]

This is mixing a bit two issues with your proposed solution. I'd see on-the-client conversion of OSM data to a tiled vector data source as a separate issue, one that would be prohibitively effortful to do actually.

[RubenKelevra]

This is mixing a bit two issues with your proposed solution. I'd see on-the-client conversion of OSM data to a tiled vector data source as a separate issue, one that would be prohibitively effortful to do actually.

I've hoped that's the easier one since there might be a converter readily available :/

I struck through those map stuff. :)

[matkoniecz]

I worry about

• download getting more expensive in heavily mapped areas.
• performance issues (especially more complicated queries)
• possibly correctness issues - how smart is the current solution? For example, what about a huge area with addr:housenumber with all its ways and nodes outside download area? Will it suppress add:housenumber quest in its current implementation?

I tested Kraków, Poland around my home whatever it will get noticeably worse - I solved many quests and it is heavily mapped.

location: https://overpass-turbo.eu/s/UBi (highway=footway limit applied as downloading full data crashes browser).

Downloading all quests available in Poland consumed 1.63 MB, doing full data download in Vespucci consumed 3.98MB. What is a good news and really surprising to me - I expected a massive difference.

[westnordost]

I had a go at this. It mostly works, but I am not 100% content with the code yet. However, following remarks so far:

• First of all, not all quests can be downloaded with this method. For example not the housenumber quest, not the oneway quest, etc. So it will work like this that the app first downloads as many as it can in the multi-download, after that, the rest via overpass.
• the OSM API does not allow both requests that cover a too large area and requests that will return too many elements. The former is easy to check for, the latter not so much. JOSM for example forwards this unpredictability to the users by saying "Area probably small enough for download". Naturally, StreetComplete needs to handle this automatically some way.
• even if the request is still within the allowed bounds of the OSM API, requests can get so large (30000+ nodes, ways etc) that the analyzing part of it starts taking really long. In one test, the download was complete in 7s, processing and rehashing it into quests took almost a minute
• So, I should also do a profiling run and see if the performance could be tweaked somewhere
• During this multi-download, a fitting icon should be shown in the download progress indicator. What kind of icon?
• After this multi-download, everything is packed with quests. This might put new or less involved users off

[westnordost]

#1901

[peternewman]

• the OSM API does not allow both requests that cover a too large area and requests that will return too many elements. The former is easy to check for, the latter not so much. JOSM for example forwards this unpredictability to the users by saying "Area probably small enough for download". Naturally, StreetComplete needs to handle this automatically some way.

Some sort of binary search?

• During this multi-download, a fitting icon should be shown in the download progress indicator. What kind of icon?

Some sort of general progress indication, or do you mean a specific icon? If the latter, what about rendering wedges of each of the quests it can fetch via this method (perhaps continually changing them), so say a quarter or sixth of road surface, and the next bit crossing, a parking quest etc. Does that make sense, or do I need to do a terrible sketch?

• After this multi-download, everything is packed with quests. This might put new or less involved users off

But is great news for regular users!

Have you considered, or would you be able to consider in the model, even if it's not coded now, being able to update the effectively local OSM data, so where you've got related quests (like "What kind of crossing is this?" and "Do these traffic lights have a button to request green light?") the act of answering the former would patch or modify the local data too (as well as queuing the change), which would mean offline users could then get the latter quest without needing to update, and for everyone it would presumably be quicker as it wouldn't need a round trip via OSM. Given you presumably know the changes you're making to a node, hopefully it wouldn't be too technically complex, but undoubtedly a future enhancement.

[westnordost]

What do you mean, binary search? That it splits the download area into 4 sub-areas if it was too large? The multi download downloads about 50 or so at once, so the slices would be very thin. Regarding the last question: No, that would be too much effort. I explained the technicalities in some other ticket. Am 26. Juni 2020 04:26:44 MESZ schrieb Peter Newman <notifications@github.com>:

…

> * the OSM API does not allow both requests that cover a too large area and requests that will return too many elements. The former is easy to check for, the latter not so much. JOSM for example forwards this unpredictability to the users by saying "Area _probably_ small enough for download". Naturally, StreetComplete needs to handle this automatically some way. Some sort of binary search? > * During this multi-download, a fitting icon should be shown in the download progress indicator. What kind of icon? Some sort of general progress indication, or do you mean a specific icon? If the latter, what about rendering wedges of each of the quests it can fetch via this method (perhaps continually changing them), so say a quarter or sixth of road surface, and the next bit crossing, a parking quest etc. Does that make sense, or do I need to do a terrible sketch? > * After this multi-download, everything is _**packed**_ with quests. This might put new or less involved users off But is great news for regular users! Have you considered, or would you be able to consider in the model, even if it's not coded now, being able to update the effectively local OSM data, so where you've got related quests (like "What kind of crossing is this?" and "Do these traffic lights have a button to request green light?") the act of answering the former would patch or modify the local data too (as well as queuing the change), which would mean offline users could then get the latter quest without needing to update, and for everyone it would presumably be quicker as it wouldn't need a round trip via OSM. Given you presumably know the changes you're making to a node, hopefully it wouldn't be too technically complex, but undoubtedly a future enhancement.

[peternewman]

What do you mean, binary search?
That it splits the download area into 4 sub-areas if it was too large?

Yeah, I wasn't sure if you're selecting an area or a circle (as a user we just get to pick the centre don't we), but if it's too big, if you select something with half the area, hopefully it would be half as big and perhaps within the API limit.

The multi download downloads about 50 or so at once, so the slices would be very thin.

That's why I suggested changing them, so maybe each segment is a fifth of the circle, and there are ten different images it cycles through, or something where they are grouped into master quest types (like the awards) and each bit has some from it's part or similar.

Regarding the last question: No, that would be too much effort. I explained the technicalities in some other ticket.

Yeah I think I read that ticket, I thought that was still when the API was doing most of the processing work. Are you not effectively now downloading an array of objects each with key-value pairs or something? Which then get processed to generate quests.

Actually scanning the source a bit, is the easy bit (like crossing->traffic lights) already done for the quests where it's possible via isApplicableTo and any additional quests would involve far more complex recreation of the Overpass API?

Or to put it another way, is it only those style quests that can be downloaded via this new multi-method anyway?

[westnordost]

Or to put it another way, is it only those style quests that can be downloaded via this new multi-method anyway?

Yes, but most quests are those simple ones.

[RubenKelevra]

Super cool to have this implemented! 😎

[RubenKelevra]

@westnordost looks like this fixed some bugs, too!

I see a lot more quests now which just didn't show up before. :)

[Harvester57]

It needs to be said: this improvement is amazing ! At first, when I looked at the benchmarks you posted, I thought "oh this looks like a nice improvement..." but it feels completely different when you try it. For rural areas, with not-so-many objects, the search is near instantaneous !

Big kudos for this !