how to improve hashlib speed with async

[scheung38]

How can I improve my hashlib performance using async await if I need to iterate and process 100,000 CSV files and hashlib for each one

Would i expect much improved performance against standard method, especially if these csv files are remotely in shared network drives?

Win 10/Linux environments

[dmontagu]

You can use starlette's run_in_threadpool function to run any expensive functions that release the GIL. It's not clear exactly what you are doing but if the problem is just that hashing so many large files is taking a long time, using run_in_threadpool might speed it up -- file IO will release the GIL, and I think the hashing functions might as well (not sure), so you should be able to achieve similar performance to what you'd get through multiprocess-based parallelism.

But it will depend a lot on what you are doing. If what you are asking about isn't specific to FastAPI this question may be better asked somewhere else like stack overflow.

[tiangolo]

Thanks @dmontagu ! 🙇‍♂️

Yeah, what @dmontagu said. If it's taking a long time, you might want to do it in a background task. Possibly in an external system like ARQ or Celery.

[github-actions[bot]]

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.