An example of deploying model using FastAPI and TritonServer

FastAPI: https://fastapi.tiangolo.com

TritonServer: https://github.com/triton-inference-server/server

To start app run:

bash build_and_run_app.sh run

To stop app run:

bash build_and_run_app.sh stop

Once the app starts running:

To check model and triton-server status:

curl localhost:8080/health

To classify an image:

curl --header "Content-Type: application/json" \
       --request POST \
       --data '{"img_ID":Key_ID of the image,"img_Path":Image save path}' \
        localhost:8080/predict

Image file reading and storage have a directory dependency (/home/ubuntu/image_data). If users have image data located in other local directories, users can change the line 56 in build_and_run_app.sh "-v /home/ubuntu/image_data:/image_data" to "-v :/image_data". API container will bind this localhost directory to container volume /image_data. When you send a request to port 8080, img_Path should be "/image_data/<NAME_OF_IMAGE>"

Save a torch script (or any other models based on any popular frameworks, TF, ONNX, Caffe, MXNet, etc.), in form of

<model_repo>

   |
   
   |____<name_of_model>
   
               |
               
               |_______config.pbtxt
               
               |
               
               |_______<1>_______model_version_1.pt
               
               |
               
               |_______<2>_______model_version_2.pt
               
               |
               
               .
               
               .
               
               .

This repo demenstrate an example of using tritonserver with grpc protocal.