Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best practice for handling rate limits #34

Closed
iserialize opened this issue Oct 13, 2022 · 1 comment
Closed

Best practice for handling rate limits #34

iserialize opened this issue Oct 13, 2022 · 1 comment

Comments

@iserialize
Copy link

iserialize commented Oct 13, 2022

I have bit of code which i have added aiometer. What iam trying to understand is how best to optimize it to get the maximum performance. The API iam calling has a rate limit of 300 requests per minute hence setting the max_per_second to 5 (if i understand this correctly) should be within the limits however, i hit 429's even then. My code is below:

async def async_iterate_paginated_api(
    function: Callable[..., Awaitable[Any]], **kwargs: Any
) -> AsyncGenerator[List[Any], None]:
    """Return an async iterator over the results of any API."""
    page_number = 1
    response = await function(**kwargs, pageNumber=page_number)
    yield response.get("entities") 
    total_pages = response.get('pageCount')
    pages = [
        page
        for page in range(2, total_pages + 1)
    ]
    async def process(function: Callable[..., Awaitable[Any]], kwarg: Any, page: Any):
        response = await function(**kwarg,  pageNumber=page)
        return response.get('entities')

    async with aiometer.amap(functools.partial(process, function, kwargs), pages, max_per_second=5) as results:
        async for result in results:
            yield result

Any feedback would be helpful.

@iserialize
Copy link
Author

With some testing, i think one of the issues may be that if i run the routine couple of times, the rate limit counter hasnt reset on the api hence subsequent runs will have less of a runway to run on. Any recommendations on how to best deal with this scenario?

Repository owner locked and limited conversation to collaborators Jan 18, 2023
@florimondmanca florimondmanca converted this issue into discussion #39 Jan 18, 2023

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant