This project benchmarks the performance of a machine learning model server under different request strategies and deployment architectures.
The goal is to analyze how different request-handling methods perform when sending concurrent or sequential HTTP requests to a model-serving endpoint. Additionally, it compares performance between a standalone server and a load-balanced setup using NGINX.
The system supports two deployment configurations:
A single FastAPI server running on port 8000.
Three FastAPI containers behind an NGINX reverse proxy that distributes load using the least_conn policy.
Running on port 80
Navigate to the requestHuggingFace folder And run docker compose
cd ./requestHuggingFace
docker compose up --build