Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overpass API 429s polygon queries #391

Closed
gboeing opened this issue Apr 24, 2017 · 1 comment
Closed

Overpass API 429s polygon queries #391

gboeing opened this issue Apr 24, 2017 · 1 comment

Comments

@gboeing
Copy link

gboeing commented Apr 24, 2017

If I query http://www.overpass-api.de/api/interpreter for streets within a fairly simple polygon, I successfully receive the data and status code 200. If I submit the same query with a slightly less-simple polygon, I receive status code 429 "too many requests" and no data. It does not matter what order these queries are posted, complex polygons always result in a 429 and simpler polygons always result in a 200 code. The API status page tells me I have slots available and adjusting my timeout changes nothing.

Most importantly, all of these queries worked for months, up until I last ran them a couple days ago. This problem only appeared this morning. I have production code using these queries to get streets within some polygon. My users are now getting 429 errors every time they try to use it.

Here is a minimal working example using Python and its requests package:

import requests
url = 'http://www.overpass-api.de/api/interpreter'
timeout = 60

# THIS QUERY FAILS WITH STATUS CODE 429 EVERY TIME
data = '[out:json][timeout:60];(way["highway"]["area"!~"yes"]["highway"!~"cycleway|footway|path|pedestrian|steps|track|proposed|construction|bridleway|abandoned|platform|raceway|service"]["motor_vehicle"!~"no"]["motorcar"!~"no"]["access"!~"private"]["service"!~"parking|parking_aisle|driveway|private|emergency_access"](poly:"37.82365 -122.24937 37.82482 -122.24812 37.82521 -122.24802 37.82624 -122.24768 37.82724 -122.24594 37.82832 -122.24426 37.82866 -122.24444 37.83009 -122.24332 37.83001 -122.24302 37.83001 -122.24282 37.83044 -122.24109 37.8305 -122.23906 37.83057 -122.2386 37.83062 -122.23789 37.8307 -122.23755 37.83084 -122.23725 37.83082 -122.23705 37.83076 -122.23692 37.83104 -122.23606 37.83173 -122.23228 37.83303 -122.22808 37.83234 -122.22705 37.83215 -122.22696 37.83208 -122.22687 37.83203 -122.22675 37.832 -122.22646 37.83193 -122.22628 37.83125 -122.22503 37.83024 -122.22449 37.82921 -122.22372 37.82761 -122.22236 37.82742 -122.22226 37.82725 -122.2221 37.82582 -122.22135 37.82355 -122.21849 37.82302 -122.21725 37.82269 -122.21662 37.8226 -122.2163 37.82251 -122.21618 37.82237 -122.21613 37.82173 -122.21549 37.82117 -122.21444 37.82012 -122.21272 37.81873 -122.21015 37.81771 -122.21161 37.81746 -122.21193 37.81704 -122.21233 37.81665 -122.2126 37.81623 -122.2128 37.81581 -122.21291 37.81506 -122.21302 37.81477 -122.21311 37.81464 -122.21319 37.81406 -122.21367 37.81359 -122.2139 37.81338 -122.21404 37.81269 -122.21486 37.81252 -122.21502 37.81228 -122.21517 37.81278 -122.21652 37.81311 -122.21829 37.81353 -122.22016 37.8138 -122.22114 37.81403 -122.22263 37.81435 -122.22401 37.8147 -122.22524 37.81547 -122.22861 37.8162 -122.23223 37.81654 -122.23328 37.8171 -122.23526 37.81732 -122.23644 37.81742 -122.23759 37.81784 -122.23931 37.8181 -122.24061 37.81855 -122.2426 37.81875 -122.24335 37.81903 -122.24419 37.81926 -122.24468 37.81984 -122.24555 37.81981 -122.24564 37.81987 -122.24575 37.82021 -122.24612 37.82073 -122.24647 37.8216 -122.24747 37.82294 -122.24934 37.82339 -122.24886 37.82365 -122.24937");>;);out;'
response = requests.post(url, data=data, timeout=timeout)
print(response.status_code, response.reason)

# THIS QUERY SUCCEEDS WITH STATUS CODE 200 EVERY TIME
data = '[out:json][timeout:60];(way["highway"]["area"!~"yes"]["highway"!~"cycleway|footway|path|pedestrian|steps|track|proposed|construction|bridleway|abandoned|platform|raceway|service"]["motor_vehicle"!~"no"]["motorcar"!~"no"]["access"!~"private"]["service"!~"parking|parking_aisle|driveway|private|emergency_access"](poly:"37.82365 -122.24937 37.82482 -122.24812 37.82624 -122.24768 37.82724 -122.24594 37.82832 -122.24426 37.82866 -122.24444 37.83009 -122.24332 37.83001 -122.24282 37.83044 -122.24109 37.8305 -122.23906 37.83062 -122.23789 37.8307 -122.23755 37.83084 -122.23725 37.83082 -122.23705 37.83076 -122.23692 37.83104 -122.23606 37.83173 -122.23228 37.83303 -122.22808 37.83234 -122.22705 37.83215 -122.22696 37.83208 -122.22687 37.83193 -122.22628 37.83125 -122.22503 37.83024 -122.22449 37.82921 -122.22372 37.82761 -122.22236 37.82725 -122.2221 37.82582 -122.22135 37.82355 -122.21849 37.82269 -122.21662 37.8226 -122.2163 37.82251 -122.21618 37.82237 -122.21613 37.82173 -122.21549 37.82117 -122.21444 37.82012 -122.21272 37.81873 -122.21015 37.81771 -122.21161 37.81746 -122.21193 37.81704 -122.21233 37.81665 -122.2126 37.81623 -122.2128 37.81477 -122.21311 37.81406 -122.21367 37.81338 -122.21404 37.81269 -122.21486 37.81228 -122.21517 37.81278 -122.21652 37.81311 -122.21829 37.8138 -122.22114 37.81403 -122.22263 37.81435 -122.22401 37.8147 -122.22524 37.81547 -122.22861 37.8162 -122.23223 37.81654 -122.23328 37.8171 -122.23526 37.81732 -122.23644 37.81742 -122.23759 37.81784 -122.23931 37.81855 -122.2426 37.81903 -122.24419 37.81926 -122.24468 37.81984 -122.24555 37.81981 -122.24564 37.81987 -122.24575 37.82021 -122.24612 37.82073 -122.24647 37.8216 -122.24747 37.82294 -122.24934 37.82339 -122.24886 37.82365 -122.24937");>;);out;'
response = requests.post(url, data=data, timeout=timeout)
print(response.status_code, response.reason)

Note that the only difference between these two queries is the polygon. The second polygon is just a slightly simplified version of the first (simplified using the douglas-peucker algorithm). Also note that both queries (i.e., the values of the data variables) work just fine in Overpass Turbo.

@gboeing gboeing changed the title overpass-api.de 429s a request even when I am not over the limit Overpass API 429s polygon queries Apr 24, 2017
@gboeing
Copy link
Author

gboeing commented Apr 25, 2017

Solution: set user-agent and referer string in Python requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant