Live API Event Loop

[ChromoX]

Hello,

I'm currently integrating DataBento into an async Python application. I noticed that in the client.py in the Live class there is a line: _loop = asyncio.new_event_loop().

Is there a reason you guys need an independent loop with this library? Or could I inject a pre-existing loop?

Thanks!

[nmacholl]

Hey there,

We launch our own event loop in a separate thread to isolate the client's networking from the application and support synchronous runtimes using a single backend. We could allow a loop to be specified. This is something we will consider adding in the future.

It isn't well documented on our site, but you can use the lower-level DatabentoLiveProtocol directly in your async application. This protocol will handle all the networking details and exposes some handy callbacks such as received_record() and received_metadata(). You would provide a factory for instances of this class when using asyncio.loop.create_connection(). Using this would essentially allow you to bypass the Live client entirely.

Let me know if you have any more questions.

Cheers,
Nick

[ChromoX]

Thanks for the pointer to the

databento-python/databento/live/protocol.py

Line 41 in 10c75b7

class DatabentoLiveProtocol(asyncio.BufferedProtocol):

.

I guess the one downside is that one loses the re-connection logic of the Live class.

[nmacholl]

I guess the one downside is that one loses the re-connection logic of the Live class.

Yes, the convenience features of the Live class would need to be re-implemented if you use this directly.

[ChromoX]

While poking around I found this class:

databento-python/databento/live/session.py

Line 282 in 10c75b7

class LiveSession:

Is this not exactly what I'm looking for? Has the ability to pass the event loop, and has reconnect policy. Is the only downside to using this object the fact it doesn't run the client in a Thread for protecting it against someone stopping the loop?

[nmacholl]

Is the only downside to using this object the fact it doesn't run the client in a Thread for protecting it against someone stopping the loop?

It contains some thread synchronization, which you don't need. You may also find that you don't need the callbacks/iteration implementation. A simpler async only design would be to push the received records into an asyncio Queue. You can mostly lift the reconnect logic if you'd like - it is rather simplistic and doesn't perform any sort of deduplication.

[ChromoX]

Thanks for the advice and pointers!