Asyncio support

[npradeep357]

Providing asyncio support natively will help vanna handle IO calls in better way.

[zainhoda]

@npradeep357 thanks for your comment. Could you say more about this suggestion? Where specifically would it help?

[andreped]

For our application we used Quart for our backend and overwrote some methods in Vanna to allow for async calls.

Is that something that could be of interest to Vanna? Shouldn't Vanna be inherently async? Or at least allow for async calls?

[npradeep357]

Okay @zainhoda,
Right now, everything in Vanna is sync, isn't it? Because of which, implementing libraries or handlers should also majorly go with sync approach or go with work around like asyncio.run() for each method individually.
Let's say I'm hosting a web application running in Fast-API and using vanna framework as backed for it.
Fast-API natively supports asyncio which brings in ease of use for parallel calls while development and runtime too. Ofcourse there are other ways in Fast-API like using background tasks but as I said, we need to do something like asyncio.run()...

The main block points in Vanna's framework are calls to DB and calls to APIs like Google, OpenAPI or even Vanna.ai for LLM calls which are all IO calls. Waiting for them to get back with responses is the main blocking point which makes the entire application to wait until the response hits.

What I suggest is making vanna asyncio natively will help in solving this issue by making these calls non-blocking and better support for libraries like asyncpg and so..

Let me know your thoughts on this.

[npradeep357]

@zainhoda, any suggestions or comments?

[Uzda20]

I'm limited with my knowledge to OpenAI API but maybe good starting point would be adding stream processing to increase UX -but I suppose that applies when Vanna is used with frontend. I agree that adding async support would increase performance. On the other hand probably Vanna could be started in container then WSGI is required. In case of gunicorn we can specify how many simultaneous connections can be estabilished so partially case with multiple connections is covered there. Ofc correct me if I'm wrong.

[vikyw89]

Having a thread looping just waiting for a response is kinda expensive to scale.

Having async would multiply the RPS many times given the same resource.

[npradeep357]

I tried the async changes in an offline version and the changes are only needed in LLM calls and Database calls

[PucaVaz]

Someone here already implement it? my problem is how I will do multiples generate_sql at the same time

[npradeep357]

Can you explain a bit more on the scenario

[PucaVaz]

like, current, you need to wait each generate sql, isn't a Asyncio function. Probably I will do the same thing was you, make an offline version and fix it on the llm calls