Improve response time latency

[marziehoghbaie]

Hi there,
I am recently using fast api for machine learning prediction.
However, the model is loaded once when the server is began, and I communicate with the database only twice before and after prediction, it seems that there is a bottleneck in my code. I tried to check the robustness of my server with sending requests by threads. When there is less than 10 request every thing is fine and the max response time is acceptable. But when I increase the number of thread to about 100, the server response time increase exponentially and it might get even 200 seconds for client to receive the response.
CAn wany body have any Idea?
I do use asysnc and await keywords in both when accessing to the database and the prediction function.
The sequential response time is less than 1 second.
How can I improve the performance of my server when it is supposed to process a huge number of requests.

[ycd]

What do you mean by improve response time latency?

[marziehoghbaie]

@ycd I provided more details about my problem

[marziehoghbaie]

The response time is quite annoying, cause every thing works fine, my friend has implemented the same project in flask, and its delay is less than 10 second under the same circumstances,
And it make me to think why FastAPI does not work?
it supposed to be faster

[ycd]

You are missing the main point of asynchronous IO.

it supposed to be faster

You are doing a heavy computation that takes more than >1 sec. If that computation is not a coroutine(It's 99% not.) asyncio does not do any context switching.

This means it is going to block your event loop.

You should use def instead of async def for that endpoint. By default, FastAPI will run handle your request run_in_threadpool when your endpoint is not a coroutine. It is better for endpoints that does heavy computation.

Can you try changing that endpoint to def instead of async def?

[marziehoghbaie]

this is my end point
@router.post('')
async def predict(x: str = Body(...), file: UploadFile = File(...),
user: Optional[User] = Depends(fastapi_users_.get_current_user)):
pass

As I understood from your message, I should remove the aysnc, But if I do this I can't use any await before DB operation of prediction function, SO I should remove them too?

[marziehoghbaie]

Besides, If one user sends many requests at once they might need to update the DB in an async manner in a update function and to avoid resource starvation I do something like

    try:
        session.add(db_item)
        session.commit()
        session.refresh(db_item)
    except:
        session.rollback()
    finally:
        session.close()

[marziehoghbaie]

You are missing the main point of asynchronous IO.

it supposed to be faster

You are doing a heavy computation that takes more than >1 sec. If that computation is not a coroutine(It's 99% not.) asyncio does not do any context switching.

This means it is going to block your event loop.

You should use def instead of async def for that endpoint. By default, FastAPI will run handle your request run_in_threadpool when your endpoint is not a coroutine. It is better for endpoints that does heavy computation.

Can you try changing that endpoint to def instead of async def?

I tried this solution, and I faced so many other issues. It seems that my model is not fully loaded and the prediction function raises an error. I loaded my model once when I create the routers, so all the requests can access it, but when I remove the async, the model can do the prediction.

[ycd]

As I understood from your message, I should remove the aysnc, But if I do this I can't use any await before DB operation of prediction function, SO I should remove them too?

Yes that's correct.

Also, you can run an asynchronous function with asyncio.run() inside a non async function.

def dummy():
    asyncio.run(my_coroutine())

[ycd]

but when I remove the async, the model can do the prediction.

What's wrong with this?

It seems that my model is not fully loaded and the prediction function raises an error.

Can you share the error message?

[marziehoghbaie]

but when I remove the async, the model can do the prediction.

What's wrong with this?

It seems that my model is not fully loaded and the prediction function raises an error.

Can you share the error message?

Yes Sure,
raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("X/mul_24:0", shape=(?, 3), dtype=float32) is not an element of this graph.
this is the error

[marziehoghbaie]

I have implemented the whole prediction process as a Class, at first I create the mentioned object, then call a load_model function in the router script, then I implement the endpoints and then call the prediction function of the created class object in each endpoint,
It works fine except for the latency problem , but when I remove the async tag, it falls apart...

As far as I know this problem relates to the tensorflow graph

[marziehoghbaie]

I have implemented the whole prediction process as a Class, at first I create the mentioned object, then call a load_model function in the router script, then I implement the endpoints and then call the prediction function of the created class object in each endpoint,
It works fine except for the latency problem , but when I remove the async tag, it falls apart...

As far as I know this problem relates to the tensorflow graph

I solved the model problem but still, the server response time is not really satisfying. When I send requests sequentially the server processing time is less than one second, but now the maximum response time is about 30 seconds, which is not really good. I should notice that I have ignored the dataBase writing and when I included again I got the following error
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
I previously have got this before and I tried to solve it by creating sessions like :
try: session.add(db_item) session.commit() session.refresh(db_item) except: session.rollback() finally: session.close()

and now, even though I have not changed the database part, and still use the same code, but it seems that there is database starvation problem

[marziehoghbaie]

This is my code::
``
x, y= asyncio.run(my_model.predict(file_path))

asyncio.run(update_db(x, y))

session = Session()

db_item = ...
try:
    session.add(db_item)
    session.commit()
    session.refresh(db_item)
except:
    session.rollback()
finally:
    session.close()

``

[tiangolo]

Thanks for the help @ycd! 🚀

@marziehoghbaie please provide a minimal, self-contained example that I (or someone else) can run here to see your problem, otherwise it's quite difficult to help, guessing what you might have in your code.

Sorry for the long delay! 🙈 I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.