You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have bit of code which i have added aiometer. What iam trying to understand is how best to optimize it to get the maximum performance. The API iam calling has a rate limit of 300 requests per minute hence setting the max_per_second to 5 (if i understand this correctly) should be within the limits however, i hit 429's even then. My code is below:
async def async_iterate_paginated_api(
function: Callable[..., Awaitable[Any]], **kwargs: Any
) -> AsyncGenerator[List[Any], None]:
"""Return an async iterator over the results of any API."""
page_number = 1
response = await function(**kwargs, pageNumber=page_number)
yield response.get("entities")
total_pages = response.get('pageCount')
pages = [
page
for page in range(2, total_pages + 1)
]
async def process(function: Callable[..., Awaitable[Any]], kwarg: Any, page: Any):
response = await function(**kwarg, pageNumber=page)
return response.get('entities')
async with aiometer.amap(functools.partial(process, function, kwargs), pages, max_per_second=5) as results:
async for result in results:
yield result
Any feedback would be helpful.
The text was updated successfully, but these errors were encountered:
With some testing, i think one of the issues may be that if i run the routine couple of times, the rate limit counter hasnt reset on the api hence subsequent runs will have less of a runway to run on. Any recommendations on how to best deal with this scenario?
Repository owner
locked and limited conversation to collaborators
Jan 18, 2023
I have bit of code which i have added aiometer. What iam trying to understand is how best to optimize it to get the maximum performance. The API iam calling has a rate limit of 300 requests per minute hence setting the max_per_second to 5 (if i understand this correctly) should be within the limits however, i hit 429's even then. My code is below:
Any feedback would be helpful.
The text was updated successfully, but these errors were encountered: