Rate limits seem to be applied after the query is performed

[ezheidtmann]

I'm using the Overpass API and have implemented an exponential backoff when I get a 429 response. But it seems like the requests take just as long for a 200 as for a 429, indicating that the server is performing a query and THEN telling me I'm rate-limited. I would expect that the server would check my rate limit before running the query, avoiding wasting those resources.

If I'm mistaken, then sorry for the noise.

Best,
Evan

[mmd-osm]

Please check /api/status to get details on your current quota, available slots and minimum wait time before issuing the next requet. Exponential backoff is just a heuristic, and does not reflect the actual state on the server. Also 429 could be related to your query taking too much time and running in a timeout.

[ezheidtmann]

Thanks! I don't think that existed when I first wrote my consumer. Is there documentation on that endpoint?

[mmd-osm]

See this post http://listes.openstreetmap.fr/wws/arc/overpass/2016-06/msg00007.html
(sympa is a bit silly, you need to click the "I'm not a spammer" button, go back to github and hit the link again)

a few days ago there has been a remark on osm-dev that is hard to figure out whether a 429 comes from the quota or a runaway request:
https://lists.openstreetmap.org/pipermail/dev/2016-June/029299.html

To fix that problem I have added an API call
/api/status
that gives to the client all information about running queries of that client.

The return syntax is

Connected as: $CLIENT_SPECIFIC_TOKEN
Rate limit: $ACCEPTED_PARALLEL_REQUESTS
$AVAILABLE slots available now.
Currently running queries (pid, space limit, time limit, start time):
... (four items per line as the headline tells)

The API call
/api/kill_my_queries
has got an upgrade as well and uses now "kill -9". I don't know when and why standard "kill" ceased to work, but "kill -9" now again does the job.

The new API call has been committed to the minor_issues branch and is already deployed on overpass-api.de

[drolbr]

The exact mechanism is as follows:

After each request the server adds extra time before it admits the next request. The length of the extra time is proportionate to the query's runtime. The factor is the higher the higher the general load is on the server. Otherwise you could block an entire CPU on the server from a single client. That would not be fair to fellow users, and the fair share depends on the number of concurrent fellow users.

A second mechanism that might cause confusion is that the server delays a request up to 15 seconds before it starts if there is not slot available yet. The server counts for the quota only the runtime after the delay. This is to facilitate to fire a bunch of requests asynchronously. If the total load is reasonable, it just works, and you do not need to implement quota management on the client side in that case.