Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAPNIK issues, how to resolve? #1286

Open
InI4 opened this Issue Mar 3, 2019 · 27 comments

Comments

Projects
None yet
5 participants
@InI4
Copy link

InI4 commented Mar 3, 2019

Issue Type

[x] Question

Description and/or steps/code to reproduce the problem

I am a bit confused. As of the MAPNIK policy issue, my app started to show empty maps lately.

  • As it was for old users with full cache only lately, I wonder how many users considered the app broken, as the app never showed a map to them?
  • I did an emergeny build to switch from MAPNIK to OpenTopo. But maps are not getting smooth lately, so I fear, they are either overloaded now, or also start to enforce policies? So sooner or later as cache expires, map will probably black out too?!
  • I tried to the 6.1.0 snapshot to get back to MAPNIK, but they to block now, I only get "forbidden" http headers from them?
  • Retrieving the old "change user agent" trick (cf. #366 ) with the new API seems to help a bit. But can it be considered relyable?

By now I am really puzzled, how get my app to work reliably again?

Version of osmdroid the issue relates to:

6.0.3

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 3, 2019

@InI4 Too many things at the same time.

  • is there a (possibly old) version of osmdroid that works OK today, MAPNIK-wise?
  • lately #1279 added the use of a Semaphore, for the access to MAPNIK only, implementing the limitation to 2 concurrent downloads (as stated in MAPNIK terms of use): did that impact your performance? In that case, could you try to remove the "2" parameter in TileSourceFactory?
  • when adding the Semaphore (only to MAPNIK so far), there can be a performance impact because we use only 2 concurrent download threads AND regardless, only 2 concurrent actual downloads. There might be cases when the threads waiting for the semaphore are "not the right ones", maybe...
  • if the map cache expires, there shouldn't be a black out: the expired bitmap is displayed, and if a better version cannot be downloaded I don't think the expired version is removed

My own tests:

  • osmdroid demo app (latest merge) seems very slow, MAPNIK-wise
  • an old app of mine that uses an old version of osmdroid seems slower than before, but still faster than osmdroid demo
  • I need to have a look at the "forbidden" error messages
@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 3, 2019

First comments after having a look at the logs:

  • using the latest merge of osmdroid demo app and MAPNIK, I never get download errors (therefore no "forbidden")
  • it looks faster than when I first tested half an hour ago: perhaps it's mainly an issue regarding MAPNIK servers currently
  • I guess the "pre-caching" of tiles is so far a bad idea or not well implemented (for the record I did it)
    • we didn't get significantly smoother performances when panning the map
    • as it works on a separate thread, technically we go beyond the concurrency limit (at least it was so before the semaphore was introduced)
    • the tile queue should be smarter, in order to discriminate between tiles that we really need and tiles that we want to pre-cache, and I'm not sure that's currently working that way
    • in first approach, I suggest to remove the pre-cache algorithm - and it means implementing a sort of MapTileProviderBasic without the getTileCache().getPreCache().addProvider( lines (or just getTileCache().getProtectedTileComputers().clear())
@InI4

This comment has been minimized.

Copy link
Author

InI4 commented Mar 4, 2019

@monsieurtanuki I apologize, I was a bit in a state, when users reported losing their maps (when there is nothing in cache, cause you are either new, or you go to a location, you haven't been before).

One more question which kind of makes it a feature request: I failed to find an API to determine, if a map provider has an issue. Background: I will add a whole bunch of map providers to my app now, to let users choose. It would be nice, if the app could tell them, there is a problem with tile loading and user should switch map tile service ... or even to switch automatically.

Some count/percentage of failed tile loads could be nice. Personal I always like exponential aging on such numbers ...

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 4, 2019

I've just sent a message to the MAPNIK guys regarding the "forbidden" tiles (cf. #1288).
I'm currently working on static tile provider stats.

@InI4

This comment has been minimized.

Copy link
Author

InI4 commented Mar 4, 2019

Digging into things, I also realized, that I once hat the calls active to change the USER-AGENT for my app. Then, after some API change, that call was not working any more and so my app did probably also run with the default user agent again, which is probably a very bad idea. (I crossposted this from #1288).

@InI4

This comment has been minimized.

Copy link
Author

InI4 commented Mar 4, 2019

@monsieurtanuki concerning the fail count, consider something like this:

double failureMeasure = 0.0;
...
// after request, "age" the old statistics
failureMeasure *= 0.9;
// If you have a classification of failures, you could also add different amounts depending on type.
if ( failure) failureMeasure += 1.0;

This gives a value between 0.0 and 10.0, where 0.0 means all requests worked, 10.0 means, infinitely many requests failed and a single failure means >= 1.0.

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 4, 2019

Actually my point is to gather static stats about the download:

  • about 30 tiles
  • being downloaded by different sources
  • how long does it take and how big are the tiles

I'm not really into your failure detection, it's too sophisticated for me.

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 4, 2019

Currently I cannot use OpenTopoMap anymore: timeout for https://a.tile.opentopomap.org/12/2103/1345.png or https://opentopomap.org/12/2103/1345.png.
Btw in TileSourceFactory it looks like the good tile url prefixes should rather be https://{a|b|c}.tile.opentopomap.org/

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 4, 2019

Maybe it's the "kiss of death": OpenStreetMap and osmdroid are so popular now that the servers crash and they need to apply stricter rules when the servers are up.

@InI4

This comment has been minimized.

Copy link
Author

InI4 commented Mar 4, 2019

A statistic about the different providers is also a great idea. It can well be, that osm infrastructure has come under pressure lately. So if you did such measures you might need to repeat them on different times of day, times of week? How long was it, when they tried to enforce better client headers? We hat these policy waves more than once.

@Firefishy

This comment has been minimized.

Copy link

Firefishy commented Mar 4, 2019

Maybe it's the "kiss of death": OpenStreetMap and osmdroid are so popular now that the servers crash and they need to apply stricter rules when the servers are up.

I'm part of the OpenStreetMap.org sysadmin team. Yes, we have blocked the unset default user-agent of osmdroid. Please set a real app user-agent eg: github-firefishy-map/0.1 and your app(s) will work again.

Please be aware we have a usage policy and because our services are so overloaded we are having to start strictly enforcing it: https://operations.osmfoundation.org/policies/tiles/

NO BULK DOWNLOADING!

@Firefishy

This comment has been minimized.

Copy link

Firefishy commented Mar 4, 2019

Linked issue from OSM site: openstreetmap/operations#281 and commit: openstreetmap/chef@bb8fbb8

@InI4

This comment has been minimized.

Copy link
Author

InI4 commented Mar 5, 2019

Thank you @Firefishy ! It is really annoying, that I forgot to fix user-agent after that API change :-)

@monsieurtanuki would it be a plan to make something like the app package name the default client name in osmdroid-android? That would help messy developers like me, that they cannot forget to do it themselves ....

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 5, 2019

@InI4 Actually I do plan to create a broader "OSM vs. osmdroid" issue - as evoked with @Firefishy in private mail exchanges.
Regarding your point, the trouble is that the user agent is already supposed to be set to the package name by default (cf. DefaultConfigurationProvider).
By curiosity, how do you manage to populate your user agent with another value? You don't use DefaultConfigurationProvider? You explicitly overwrite the user agent value?

@Firefishy

This comment has been minimized.

Copy link

Firefishy commented Mar 5, 2019

@monsieurtanuki Please drop the "OSM vs. osmdroid" wording. OpenStreetMap has extremely limited resources (admins, servers, bandwidth) and all not for profit... The block on the osmdroid user-agent is because apps using the library are not following our policy. The policy is to keep our services from grinding to halt. Last week osmdroid was the highest non-browser user-agent and services in some regions were severely degraded (see linked issues above).

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 5, 2019

@Firefishy Oh, I get your point: I didn't mean "against", I just wanted to give a temporary short title with both names osmdroid and OSM. I rather meant something like "osmdroid and a correct use of OSM resources". Apparently "vs." has too aggressive a meaning, my translation into English was not correct.
Sorry about that.

@monsieurtanuki monsieurtanuki referenced this issue Mar 6, 2019

Closed

Correct use of OSM resources #1289

1 of 1 task complete
@InI4

This comment has been minimized.

Copy link
Author

InI4 commented Mar 7, 2019

@monsieurtanuki my bad, the DefaultConfigurationProvider code I see, is nearly what I was trying to suggest: It retrieves the package name. So only an minor suggestion would be to attach a "/"+PackageInfo.versionCode as suggested by @Firefishy above. As this is suggested, this would help an app to get access again after an update. Only bad for the OSM guys, when you didn't fix the DOS aspect ..

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 7, 2019

@InI4 With #1289

  • you'll get a more explicit message about setting the user agent for all the tile sources that explicitly demand a meaningful user agent value
  • the OSM servers won't even have to reply with a "403 forbidden" to requests that won't be sent to them.

As for the value of the user agent, should it include the version number? Unless there's an explicit need expressed from OSM, I guess we can skip the version code part for the moment. Anyway we're talking about an init default value for the user agent, that developers can overwrite whenever they want.

@monsieurtanuki monsieurtanuki self-assigned this Mar 7, 2019

@Firefishy

This comment has been minimized.

Copy link

Firefishy commented Mar 7, 2019

I recommend including the version number in the user-agent. eg: appname/0.1
That way if there is a faulty version of an app which caused problems, it could be filtered instead of filtering all versions of the app.

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 7, 2019

Fair enough. Unfortunately we (in osmdroid) set a default user agent only when the app is launched for the very first time. Then the value is kept and not automatically updated.
Therefore, in the best situation the user agent will always be the package name and the version number of the first installation. No version number change will be sent to OSM.

An alternate solution: if we agree that OSM explicitly demands something like
context.getPackageName() + "/" + PackageInfo.versionCode
we can force this value for the user agent sent to OSM, regardless of the user agent value that we (in osmdroid) computed and used so far.

@Firefishy Does that make sense?

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 8, 2019

package name + version number: just PR'ed in #1293.

@ambrose-at-cordic

This comment has been minimized.

Copy link

ambrose-at-cordic commented Mar 13, 2019

I have an app which was being blocked as it was using a unique string rather than the package name for the user-agent value,

Once I used: OpenStreetMapTileProviderConstants.setUserAgentValue(getApplicationContext().getPackageName()): instead, that resolved the problem. The problem is, when our tested tried reporduce the problem on the same device with version of the app without the aforemenioned change, it wasn't blocked. Why would this happen? shouldn't it have blocked it once it had seen the invalid user-agent value?

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 13, 2019

If I remember well

  • "osmdroid" was the default user agent in osmdroid
  • osmdroid users/developers were supposed to change it
  • OSM servers experienced too heavy a load, and they decided to block osmdroid developers that didn't change the user agent
  • they did that by blocking requests whose user agent is/starts with "osmdroid"

Perhaps:

  • your user agent string was/started with "osmdroid"
  • in osmdroid in some cases the user agent was computed at init time only, then stored in the prefs - your testers may or may not have already something in the prefs, which would prevent overriding the user agent value

This is my best guess.

@ambrose-at-cordic

This comment has been minimized.

Copy link

ambrose-at-cordic commented Mar 14, 2019

I went back and looked at the logcat output for the version of the app without the user agent fix (cf. #366 ), and it was blocking requests for the tiles, but still showing tiles for places that I had previously viewed but if I went to other places I hadn't been before, nothing was shown which means that those tiles were cached, which gives me more confidence in the user agent fix, when I was using that version of the app, it was a re-install, which should have removed the local tile cache, so where would have those intial tile images come from? any ideas?

@monsieurtanuki

This comment has been minimized.

Copy link
Collaborator

monsieurtanuki commented Mar 15, 2019

@ambrose-at-cordic The reinstall doesn't necessarily clear the tile cache, as it is (by default I think) directly in /osmdroid (I mean, not in /Android/data/...). For historical reasons (cf. #1233).
Therefore, if the tile cache is not cleared, that would explain that you see cached tiles from one install to the other.
If you want to perform a clear reinstall, think about deleting /osmdroid/tiles/cache.db (or something like that).

@spyhunter99

This comment has been minimized.

Copy link
Collaborator

spyhunter99 commented Apr 4, 2019

is this still a problem?

@ambrose-at-cordic

This comment has been minimized.

Copy link

ambrose-at-cordic commented Apr 9, 2019

No the maps have been working with no issues since then. From what I remember of the issue, I confirmed that for the version prior to the fix, any tiles that were accessible prior the block could still be viewed but if you went a place you hadn't been before, you wouldn't get the tiles, but in version with the fix, you would get those tiles, so it the fixed resolved the issue for us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.