A framework for deploying serializable and optimizable neural net models at scale in production via. the NVIDIA Triton Inference Server.
Documentation - Guides, Models, API References
- Data Processing
- Intuitive data processing modules for encoding human-readible data into tensors compatible with deep learning libraries
- Model Exporting
- FastNN torch modules and tools for exporting models via.
TorchScript
tracing and scripting to a production environment. Now includes Text Generation.
- FastNN torch modules and tools for exporting models via.
- Model Zoo
- Various exported models hosted in this repo via. git-lfs and AWS S3. Includes models from the HuggingFace's Transformers and TorchVision
- Model Deployment
- Deploy models using Triton Inference Server on CPU/GPU-compatible server(s) with helm or docker
- FastNN Client
- Client wrapper for Triton Inference Server's client module for programmatic requests with python
Git LFS is required if you'd like to use any of the models provided by FastNN in ./model_repository
.
Cloning this repository without Git LFS will clone a repository with LFS pointers, not the actual model.
After the repository is cloned and Git LFS is installed, set up with git lfs install
.
Download specific models with:
git lfs pull --include="model_repository/<path-to-model-dir>" --exclude=""
Download ALL models with:
git lfs pull
# Or
#git lfs pull --include="*" --exclude=""
Requirements: Python 3.9+, PyTorch 2+, Triton Client
Optional: CUDA Compatible GPU, NVIDIA Drivers, cudnn (PyTorch pre-built wheels)
-
To install PyTorch with TorchVision, please refer to the installation instructions on their web page here.
-
The tritonclient package wheels are not hosted on the public PyPI server. We need to add the address of NVIDA's private python package index to the environment. You can complete these steps and install the tritonclient package by running the following.
# If you cloned this repo, you can just uncomment and run the one line below
#sh ./scripts/install_triton_client.
pip install nvidia-pyindex
pip install tritonclient[all]
Once the above requirements are set, you can install fastnn with the command below:
pip install fastnn
If you are comfortable with the latest default stable releases of PyTorch you can skip step 1 in the pre-requisites and run pip install fastnn[torch]
instead.
You will need to install poetry by referring to the installation instructions on their web page here.
After cloning the repository, just run poetry install
to install with the .lock file.
Activate the virtual environment with the command below:
poetry shell
Official FastNN images are hosted on Docker Hub.
Select FastNN package and image versions by referencing the documentation. Development and runtime environments are available.
Jupyter lab and notebook servers are accessible with notebook examples and terminal access localhost:8888
with every image.
Run the latest FastNN image by running below:
docker run --gpus all --rm -it -p 8888:8888 aychang/fastnn:latest
Run images with specific configurations as can see in the example command below:
docker run --gpus all --rm -it -p 8888:8888 aychang/fastnn:0.1.0-cuda11.0-runtime-ubuntu18.04-py3.7
Requirements:
- Docker 19.03+
GPU Inference Requirements:
-
NVIDIA CUDA-Compatible GPU
Local deployment of the Triton Server uses the EXPLICIT model control mode. Local models must be explicitly specified with the --load-model
argument in ./scripts/start_triton_local.sh
export MODEL_REPOSITORY=$(pwd)/model_repository
sh ./scripts/start_triton_local.sh
Requirements: kubectl 1.17+, Helm 3+, Kubernetes 1.17+
You can currently install the local FastNN helm chart with the following instuctions:
cd ./k8s
helm install fastnn .
export MODEL_REPOSITORY=$(pwd)/model_repository
Note: The current local helm chart installation deploys Triton using the NONE model control mode. All models available in the S3 Model Zoo will be deployed...good luck. Deployed models and model control mode can be edited in the helm chart deployment configuration file.
This project is licensed under the terms of the MIT license.