Overpass API 429s polygon queries

[gboeing]

If I query http://www.overpass-api.de/api/interpreter for streets within a fairly simple polygon, I successfully receive the data and status code 200. If I submit the same query with a slightly less-simple polygon, I receive status code 429 "too many requests" and no data. It does not matter what order these queries are posted, complex polygons always result in a 429 and simpler polygons always result in a 200 code. The API status page tells me I have slots available and adjusting my timeout changes nothing.

Most importantly, all of these queries worked for months, up until I last ran them a couple days ago. This problem only appeared this morning. I have production code using these queries to get streets within some polygon. My users are now getting 429 errors every time they try to use it.

Here is a minimal working example using Python and its requests package:

import requests
url = 'http://www.overpass-api.de/api/interpreter'
timeout = 60

# THIS QUERY FAILS WITH STATUS CODE 429 EVERY TIME
data = '[out:json][timeout:60];(way["highway"]["area"!~"yes"]["highway"!~"cycleway|footway|path|pedestrian|steps|track|proposed|construction|bridleway|abandoned|platform|raceway|service"]["motor_vehicle"!~"no"]["motorcar"!~"no"]["access"!~"private"]["service"!~"parking|parking_aisle|driveway|private|emergency_access"](poly:"37.82365 -122.24937 37.82482 -122.24812 37.82521 -122.24802 37.82624 -122.24768 37.82724 -122.24594 37.82832 -122.24426 37.82866 -122.24444 37.83009 -122.24332 37.83001 -122.24302 37.83001 -122.24282 37.83044 -122.24109 37.8305 -122.23906 37.83057 -122.2386 37.83062 -122.23789 37.8307 -122.23755 37.83084 -122.23725 37.83082 -122.23705 37.83076 -122.23692 37.83104 -122.23606 37.83173 -122.23228 37.83303 -122.22808 37.83234 -122.22705 37.83215 -122.22696 37.83208 -122.22687 37.83203 -122.22675 37.832 -122.22646 37.83193 -122.22628 37.83125 -122.22503 37.83024 -122.22449 37.82921 -122.22372 37.82761 -122.22236 37.82742 -122.22226 37.82725 -122.2221 37.82582 -122.22135 37.82355 -122.21849 37.82302 -122.21725 37.82269 -122.21662 37.8226 -122.2163 37.82251 -122.21618 37.82237 -122.21613 37.82173 -122.21549 37.82117 -122.21444 37.82012 -122.21272 37.81873 -122.21015 37.81771 -122.21161 37.81746 -122.21193 37.81704 -122.21233 37.81665 -122.2126 37.81623 -122.2128 37.81581 -122.21291 37.81506 -122.21302 37.81477 -122.21311 37.81464 -122.21319 37.81406 -122.21367 37.81359 -122.2139 37.81338 -122.21404 37.81269 -122.21486 37.81252 -122.21502 37.81228 -122.21517 37.81278 -122.21652 37.81311 -122.21829 37.81353 -122.22016 37.8138 -122.22114 37.81403 -122.22263 37.81435 -122.22401 37.8147 -122.22524 37.81547 -122.22861 37.8162 -122.23223 37.81654 -122.23328 37.8171 -122.23526 37.81732 -122.23644 37.81742 -122.23759 37.81784 -122.23931 37.8181 -122.24061 37.81855 -122.2426 37.81875 -122.24335 37.81903 -122.24419 37.81926 -122.24468 37.81984 -122.24555 37.81981 -122.24564 37.81987 -122.24575 37.82021 -122.24612 37.82073 -122.24647 37.8216 -122.24747 37.82294 -122.24934 37.82339 -122.24886 37.82365 -122.24937");>;);out;'
response = requests.post(url, data=data, timeout=timeout)
print(response.status_code, response.reason)

# THIS QUERY SUCCEEDS WITH STATUS CODE 200 EVERY TIME
data = '[out:json][timeout:60];(way["highway"]["area"!~"yes"]["highway"!~"cycleway|footway|path|pedestrian|steps|track|proposed|construction|bridleway|abandoned|platform|raceway|service"]["motor_vehicle"!~"no"]["motorcar"!~"no"]["access"!~"private"]["service"!~"parking|parking_aisle|driveway|private|emergency_access"](poly:"37.82365 -122.24937 37.82482 -122.24812 37.82624 -122.24768 37.82724 -122.24594 37.82832 -122.24426 37.82866 -122.24444 37.83009 -122.24332 37.83001 -122.24282 37.83044 -122.24109 37.8305 -122.23906 37.83062 -122.23789 37.8307 -122.23755 37.83084 -122.23725 37.83082 -122.23705 37.83076 -122.23692 37.83104 -122.23606 37.83173 -122.23228 37.83303 -122.22808 37.83234 -122.22705 37.83215 -122.22696 37.83208 -122.22687 37.83193 -122.22628 37.83125 -122.22503 37.83024 -122.22449 37.82921 -122.22372 37.82761 -122.22236 37.82725 -122.2221 37.82582 -122.22135 37.82355 -122.21849 37.82269 -122.21662 37.8226 -122.2163 37.82251 -122.21618 37.82237 -122.21613 37.82173 -122.21549 37.82117 -122.21444 37.82012 -122.21272 37.81873 -122.21015 37.81771 -122.21161 37.81746 -122.21193 37.81704 -122.21233 37.81665 -122.2126 37.81623 -122.2128 37.81477 -122.21311 37.81406 -122.21367 37.81338 -122.21404 37.81269 -122.21486 37.81228 -122.21517 37.81278 -122.21652 37.81311 -122.21829 37.8138 -122.22114 37.81403 -122.22263 37.81435 -122.22401 37.8147 -122.22524 37.81547 -122.22861 37.8162 -122.23223 37.81654 -122.23328 37.8171 -122.23526 37.81732 -122.23644 37.81742 -122.23759 37.81784 -122.23931 37.81855 -122.2426 37.81903 -122.24419 37.81926 -122.24468 37.81984 -122.24555 37.81981 -122.24564 37.81987 -122.24575 37.82021 -122.24612 37.82073 -122.24647 37.8216 -122.24747 37.82294 -122.24934 37.82339 -122.24886 37.82365 -122.24937");>;);out;'
response = requests.post(url, data=data, timeout=timeout)
print(response.status_code, response.reason)

Note that the only difference between these two queries is the polygon. The second polygon is just a slightly simplified version of the first (simplified using the douglas-peucker algorithm). Also note that both queries (i.e., the values of the data variables) work just fine in Overpass Turbo.

[gboeing]

Solution: set user-agent and referer string in Python requests.