Why do FastAPI Insertion queries take Extremely Long (10+mins for 100,000 rows)

[sking8484]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

@router.post('/')
def postStockData(data:List[pydanticModels.StockPrices], raw_db = Depends(get_raw_db)):

  cursor = raw_db[0]
  cnxn = raw_db[1]

  
  SQL = "INSERT INTO " + models.StockPricing.__tablename__ + " VALUES (%s, %s, %s)"
  
  valsToInsert = []
  for row in data:
    rowD = row.dict()
    valsToInsert.append((rowD['date'], rowD['symbol'], rowD['value']))
  cursor.executemany(SQL, valsToInsert)
  cnxn.commit()

Description

The above code takes extremely long to run, so much so that I can not complete it.
I've tested it with SqlAlchemy, as well as a raw connector, as above, and through fastApi they both don't finish.
Running this query locally takes about 1-3 seconds to complete.
Why does it take so long?

Operating System

macOS

Operating System Details

No response

FastAPI Version

0.81

Python Version

3.10.6

Additional Context

No response

[Garito]

Bulk insert?
https://stackoverflow.com/questions/12365992/sql-insert-multiple-rows

[odiseo0]

FastAPI doesn't do insertion queries

[crocodilered]

This is not FastAPI issue but common data management problem.

Simple methods to resolve it: remove indexes from table before inserting, do not insert all the rows with one query, use DB-native tools (like prepare statement in postgres).

Also u can run the task in separate process to prevent user waiting its ending

[sking8484]

Thanks for your responses! Ended up using an insert statement with multiple value rows, as opposed to an execute many. Closing this now that it is resolved!