add async option to REST API using aio-http

[yehoshuadimarsky]

Here's an implementation of adding async capability to the REST API. Although, when I tested it it was NOT any faster for some reason, is that a server side limitation?

Closes #56

Thanks.

[takos22]

I think the code you wrote is pretty good and I also need this feature, so I approve the review. I hope this will make maintainers merge the PR.

[takos22]

Maybe you could do the following to use the dict.get default argument:

params_copy = {name: bool_map.get(value, value) for name, value in params.items()}

[pssolanki111]

I believe the new asyncio API is a good addition. Approving the PR to allow auto merge. Would be nicer if you could implement the change suggested on the PR in an earlier review.

[yjang43]

Hi, I hope to see this async option officialized in the project. Do you know when will this be added?

[pssolanki111]

@yjang43 sadly this lib seems to be largely unmaintained. If you need support for both asyncio and callback based websocket streaming library, I'm writing a Library for polygon APIs (click to check out). Although it already has most of the functionalities, all rest endpoints (stocks, references, forex, crypto), and all websocket streaming with both callback and asyncio based streaming client,
just that it doesn't have a documentation yet (I started writing this a few days ago). But I have very detailed code comments and docstrings which should make it easy to understand. installable by

pip install polygon

[yjang43]

@pssolanki111 Thanks for the suggestion. I took a look at it and the project looks great, but since my main plan is to retrieve with REST API only, I think your project does not suffice my goal, as it seems only to support websocket. Or am I missing something?

[pssolanki111]

@yjang43 Hey Yoo Sung,

Do you mean you need REST endpoints but with asyncio support? IF that is the case, I can add asyncio support using httpx. Currently the rest endpoints are implemented through requests. Now that you say, I think it would be a good addition to add to the lib to allow people having an option to both.

That's exactly what I had in mind when I decided to implement both asyncio and callback on websockets. Also added support for reconnecting the stream with only one argument changed to True.

Stay Tuned :)

[yjang43]

@pssolanki111 Thank you!
It will be so neat to have a toggle option to use async and definitely I would like to integrate yours once the option is available. Do you have an approximate date to add that functionally to REST API?

[pssolanki111]

@yjang43 Oh yes I do 😆

Here take a look: https://github.com/pssolanki111/polygon/blob/main/polygon/stocks/client.py#L507

stocks and reference endpoints are already completed. I'll prolly also get crypto and forex done before I sleep today.

The toggle is simple,

normal_client = PolygonClient(api_key)

async_client = PolygonClient(api_key, use_async=True)

All clients support context managers, (and yes support for async with is also available).

Lemme know what you think :)

[pssolanki111]

@yjang43 Hey Yoo Sung,

I just released a test version 0.0.2 of the lib. It has async support, context managers (both sync and async), and all endpoints.

install is simply pip install polygon

Please have a look and let me know :)

[pssolanki111]

@yjang43 The library is out now!!

documentation: https://polygon.readthedocs.io/en/latest/

section about async support: https://polygon.readthedocs.io/en/latest/Getting-Started.html#async-support-for-rest-endpoints

pypi page: https://pypi.org/project/polygon/

The README has 4 examples. Second example shows how to use async for rest: https://github.com/pssolanki111/polygon

more examples will be added as i get time.

[clickingbuttons]

Hey @yehoshuadimarsky, thanks for being interested enough in our Python client to add async support! We've released a new client (v1.0.0b) that uses urllib3 directly. While urllib3 doesn't have an async interface you can wrap async around our functions to see most of the gains of asyncio.

Feel free to open any issues that you may have doing that and we'll work to address them.

[schmidt-jake]

While urllib3 doesn't have an async interface you can wrap async around our functions to see most of the gains of asyncio.

@clickingbuttons can you say more about this? My understanding of python concurrency is shaky at best, but it looks like the polygon python REST client is implemented with synchronous urllib3 APIs, so I don't understand how users wrapping functions in async would actually achieve concurrent execution. It seems to me that the client would need to be (re-)written with async calls first. Btw, it looks like urllib3 does have such an async API now.

Are y'all aware of this project? It looks like someone already tried to write an async python lib for your REST API.

[clickingbuttons]

so I don't understand how users wrapping functions in async would actually achieve concurrent execution.

See the official docs.

Btw, it looks like urllib3 does have such an async API now.

Would be interesting to benchmark against the current PoolManager. I no longer work on this library, but to those who do, it may be worth checking out.

Are y'all aware of this project?

We were when I wrote it.