huggingface_resnet_pytorch

HuggingFace ResNet50 PyTorch Model

Overview

The example presents a HuggingFace ResNet50 PyTorch model inference.

Example consists of following scripts:

• install.sh - install additional dependencies for downloading model from HuggingFace
• server.py - start the model with Triton Inference Server
• client.py - execute HTTP/gRPC requests to the deployed model

And configurations:

• kubernetes - example Helm Charts for serving and test inference in Kubernetes cluster

Running example locally

To run example locally the torch package is required. It can be installed in your current environment using pip:

pip install torch

Or you can use NVIDIA PyTorch container:

docker run -it --gpus 1 --shm-size 8gb -v {repository_path}:{repository_path} -w {repository_path} nvcr.io/nvidia/pytorch:23.07-py3 bash

If you select to use container we recommend to install NVIDIA Container Toolkit.

Follow the step-by-step guide to execute the example:

1. Install PyTriton following the installation instruction
2. Install the additional packages using install.sh

./install.sh

3. In current terminal start the model on Triton using server.py

./server.py

4. Open new terminal tab (ex. Ctrl + T on Ubuntu) or window
5. Go to the example directory
6. Run the client.py to perform queries on model:

./client.py

Running example on Kubernetes cluster

The following prerequisites must be matched to run the example:

• Kubernetes cluster with NVIDIA GPU node
• NVIDIA Device Plugin installed in Kubernetes cluster
• Docker Containers Registry accessible from Kubernetes cluster
• Installed Helm for creating the deployment and test job

Optionally you may install NVIDIA Container Toolkit and NVIDIA GPU Operator which enable more features like MIG or Time Slicing support in the cluster. To learn more how to set up Kubernetes cluster with NVIDIA GPU you can review NVIDIA Cloud Native Documentation

Below, we present a step-by-step guide assuming that all the commands are executed from the root of repository.

Follow these steps to run and test example in the cluster:

1. [Optional] Build PyTriton wheel following the build instruction
2. Prepare the tag under which image is going to be pushed to your Docker Containers Registry accessible from Kubernetes cluster. Example for local cluster (minikube, k3s) with registry hosted inside the cluster:

export DOCKER_IMAGE_NAME_WITH_TAG=localhost:5000/resnet-pytorch-example:latest

3. Build and push the Docker container image to your registry:

# Export the base image used for build
export FROM_IMAGE_NAME=nvcr.io/nvidia/pytorch:23.07-py3
./examples/huggingface_resnet_pytorch/kubernetes/build_and_push.sh

Note: By default the container is built using pytriton package from GitHub. To build container with wheel built locally use export BUILD_FROM=dist before executing script.

4. Install the Helm Chart with deployment and service:

helm upgrade -i --set deployment.image=${DOCKER_IMAGE_NAME_WITH_TAG} \
resnet-pytorch-example \
./examples/huggingface_resnet_pytorch/kubernetes/deployment

5. Install the Helm Chart with client test

helm install --set image=${DOCKER_IMAGE_NAME_WITH_TAG} \
resnet-pytorch-example-test \
./examples/huggingface_resnet_pytorch/kubernetes/test

Now, you can review the logs from the running PODs to verify the inference is running. To show the logs from cluster for given POD first list all running pods:

kubectl get pods

Next show logs from server or client:

kubectl logs {NAME}

To remove the installed charts simply run:

helm uninstall resnet-pytorch-example-test
helm uninstall resnet-pytorch-example