Understanding Open Library API Rate Limits and Scaling

[Vanals]

Question

Hi,

I’m curious about how the system handles situations where an Open Library user makes “too many requests” and potentially hits a limit. If this is a thing, how does the process typically evolve?

I’ve read that you might reach out to the user—would that be to request a reduction in usage, simply to understand what’s happening (e.g., ruling out a DDoS attack), or to discuss options like a contract or donation to help support the system?

I’m asking to better understand how the API scales.

Thanks!

Best,

[mekarpeles]

Hi @Vanals our APIs don't scale terribly well and the API layer is served by the same application as the system that serves the front-end to our patrons, so when possible, we request that partners and patrons use data dumps for any processing they need to do in bulk and/or to cache results where possible.

The user-agent is used for rate limiting some bots and also gives us a way to contact services if we notice behavior where there may be better solutions (with discussion).

The Search API (/search.json) can often be used to pull work and edition level records for hundreds of books at a time (as opposed to e.g. the books or query API).

If we notice lots of requests from certain networks, then we rate limit and sometimes block entirely. Please feel free to keep us in the loop. Our policy is to prioritize patrons and end-users while offering the best support we can to library partners and other public good book services in the ecosystem.