Performance test not yielding good result when connected to Oracle DB using SQLAlchemy

[nabarunn]

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

db_connections.py

db_url = URL.create("oracle+oracledb", username="username", password="password", host="host", 
                     port=port, database="testdb",)
engine = create_engine(db_url, label_length=5, max_overflow=10, pool_size=20)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
base = declarative_base()

def db_session():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()


books_crud.py

def get_available_books(db: Session):
    return db.query(models.BooksMaster).filter(models.BooksMaster.available == 1).all()


routers.py

@router.get("/books")
async def get_available_books(db:Session = Depends(db_session)):
        available_books = books_crud.get_available_books(db)
        return available_books

Description

I have used SQLAlchemy and connected to Oracle DB with max_pool = 20 implementing the conventional method as given in code snippet. The endpoint performs a simple retrieve operation. I did a performance test with peak_concurrency = 100users, spawn_rate = 10users/second for 2min. The average response time = 2sec and throughput = 30TPS. Also db connections got exhausted and requests failed.

The expectation from this discussion would be to get an approach to connect to oracle db using SQLAlchemy ORM that provides the best performance. I have gone through multiple github discussions but were unable to find the standard or the best practice.

Operating System

Linux

Operating System Details

RHEL 8
MAC Monterey v12.6.4

FastAPI Version

0.95.1

Python Version

python 3.9

Additional Context

No response

[notypecheck]

Hello, it appears that you're using synchronous sqlalchemy API with async endpoint, it would be better if you use asynchronous sqlalchemy API or make your endpoint function synchronous (but it would lead to extra costs due to that function running in a thread executor)

Also you should use new SQLAlchemy 2.0 select API instead of legacy Session.query

You can also simplify your dependency function:

def db_session():
    with SessionLocal() as session:
        yield session

[nabarunn]

Hi @ThirVondukr thanks for the reply. I tried with the simplified dependency and for that too results are similar. For the asynchronous sqlalchemy API I am not finding proper implementation with cx_Oracle_async. And with cx_Oracle the create_async_engine will not work. I then have to write native queries without using the SQLAlchemy ORM

[PancakePuncher]

In my experience the ORM Library and the Engine you use to connect to the database can be independently split. But you will need to change how you create your functions that perform operations on your database.

Oracle is a proprietary database so I'm not able to really find many libraries that give full support for it outside of SQL Alchemy 2.0.

Peewee is another very popular ORM library for Python and I can find a discussion from 2018 where the creator says due to Oracle being a proprietary database he will not give direct support for it, but he says that it's entirely possible to sub-class of Peewee's Database object and build your own implementation and support for OracleDB.

The simplest solution might be that you are going to have to give your engine a lot more overhead.

But if you have the option to not use OracleDB I'd really recommend you use a more wildly used SQL Database like Prostgres, Maria, SQLite, etc.

If that is not an option and you wish to attempt to implement cx_Oracle_sync with SQL Alchemy ORM you might have to write you own interface that knows how to pass the ORM objects into cx_Oracle_sync

If you get it working you could maybe even publish it for other people to use.

I know this has been likely not helpful, but I spent some time really digging around and I just don't think Oracle is widely supported in the Python space.

Best of luck!