don't attempt to download further tiles if google blocked you

[flixr]

When trying to download map tiles and you get a ERROR 403: Forbidden. it should abort trying to download the rest as well. Since then you are temporarily blocked by Google and there is no point in trying to download the rest. Also print a more descriptive error message saying that you are probably temporarily blocked by google

see sw/lib/ocaml/gm.ml

[flixr]

It seems there is no really easy way to do this when using wget (as we do in sw/lib/ocaml/http.ml)
You can't distinguish which http error you got from the wget exit code, so we would have to look through the Failure string to check if the tile was not found (404) and we want to try a lower resolution tile or if we were blocked (403) and can stop trying for now....

[OpenUAS]

maybe add --server-response as in example wget --server-response http://localhost/nx.file 2>&1 | awk '/^ HTTP/{print $2}'
but then not awk but ocaml ofcouse. maybe wget returncode: 8 "Server issued an error response" will help.

Or else maybe a real a http lib from ocaml e.g. http://ocaml-http.forge.ocamlcore.org/examples/basic_auth.ml.

[flixr]

@gautierhattenberger I tried to quickly fix this by using the Http_client from ocamlnet
See my branch try_ocaml_netclient, or directly commit flixr/paparazzi@b127a4d
I didn't really get it working this evening, but you get the idea.... don't really have more time to spend on this right now.

[gautierhattenberger]

@flixr I compiled your code this morning and it seems to work like a charm. I tried to reach the download limit by getting a lot of tiles from GM, but couldn't in the end... (the limit seems higher than what I thought, or did they increased it recently ? )

[flixr]

Two reasons it didn't seem to work properly for me:

1. I screwed up the maps makefile (is now fixed again) and was wondering why I got huge tiles.
2. Forgot to delete erroneous files.

This is fixed in the pull request. Would be good if you could have a look at it though...

I also just tried to reach the limit again, was unable to reach it... so this is untested.
Will have to try at work again, there I was able to reach the limit a few days ago.

[flixr]

Regarding the google maps limit resulting in a 403:
I could quickly get into the limit again by using the old code and downloading a few tiles, switching to the new netclient implementation I could then continue to get tiles. So my guess is that this has something to do with the different user_agent of wget/netclient...

[flixr]

partially fixed with #276

[esden]

+1 and also, maybe there is a way of downloading the tiles without risking to be blocked by google?

[flixr]

short answer: no
But as mentioned above, with the ocaml netclient already merged into current master with 276) I wasn't able to run into the limit anymore, just switching back to old code, I was already in the limit, back to new code: was able to continue downloading tiles for like 20min

[flixr]

closing, as it doesn't seem to occur anymore with the new code....