Skip to content

Commit

Permalink
docs: add model support (#784)
Browse files Browse the repository at this point in the history 
* docs: add model support

* docs: fix name

* docs: clean models names

* fix: polish model and weight names

* docs: revision

* docs: revision

* docs: update benchmark table

* docs: update benchmark table and narratives

* docs: refactor ref

* docs: refactor ref

* docs: typo

* docs: typo

* docs: fix link

Co-authored-by: numb3r3 <wangfelix87@gmail.com>
Co-authored-by: ZiniuYu <ziniuyu@gmail.com>
  • Loading branch information
@shan-mx @numb3r3 @ZiniuYu
3 people committed Aug 2, 2022
1 parent 8b8082a commit c67a7f5
Showing 1 changed file with 34 additions and 19 deletions.
53 changes: 34 additions & 19 deletions docs/user-guides/server.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ This chapter introduces the API of the server.
You will need to install server first in Python 3.7+: `pip install clip-server`.
```

(start-server)=
## Start server


Expand Down Expand Up @@ -59,25 +58,41 @@ One may wonder where is this `onnx-flow.yml` or `tensorrt-flow.yml` come from. M

The procedure and UI of ONNX and TensorRT runtime would look the same as Pytorch runtime.

## Model support

Open AI has released 9 models so far. `ViT-B/32` is used as default model in all runtimes. Due to the limitation of some runtime, not every runtime supports all nine models. Please also note that different model give different size of output dimensions. This will affect your downstream applications. For example, switching the model from one to another make your embedding incomparable, which breaks the downstream applications. Below is a list of supported models of each runtime and its corresponding size. We include the disk usage (in delta) and the peak RAM and VRAM usage (in delta) when running on a single Nvidia TITAN RTX GPU (24GB VRAM) for a series of text and image encoding tasks with `batch_size=8` using PyTorch runtime.

| Model | PyTorch | ONNX | TensorRT | Output Dimension | Disk Usage (MB) | Peak RAM Usage (GB) | Peak VRAM Usage (GB) |
|----------------|---------|------|----------|------------------|-----------------|---------------------|----------------------|
| RN50 | ✅ | ✅ | ✅ | 1024 | 256 | 2.99 | 1.36 |
| RN101 | ✅ | ✅ | ✅ | 512 | 292 | 3.51 | 1.40 |
| RN50x4 | ✅ | ✅ | ✅ | 640 | 422 | 3.23 | 1.63 |
| RN50x16 | ✅ | ✅ | ❌ | 768 | 661 | 3.63 | 2.02 |
| RN50x64 | ✅ | ✅ | ❌ | 1024 | 1382 | 4.08 | 2.98 |
| ViT-B/32 | ✅ | ✅ | ✅ | 512 | 351 | 3.20 | 1.40 |
| ViT-B/16 | ✅ | ✅ | ✅ | 512 | 354 | 3.20 | 1.44 |
| ViT-L/14 | ✅ | ✅ | ❌ | 768 | 933 | 3.66 | 2.04 |
| ViT-L/14@336px | ✅ | ✅ | ❌ | 768 | 934 | 3.74 | 2.23 |

### Use custom model
## Model support

You can also use your own model in ONNX runtime by specifying the model name and the path to model directory in YAML file.
The various `CLIP` models implemented in the [OpenAI](https://github.com/openai/CLIP), [OpenCLIP](https://github.com/mlfoundations/open_clip), and [MultilingualCLIP](https://github.com/FreddeFrallan/Multilingual-CLIP) are supported.
Due to the limitation of some runtimes, not every runtime supports all models.
Please also note that **different models give different sizes of output dimensions**. This will affect your downstream applications. For example, switching the model from one to another make your embedding incomparable, which breaks the downstream applications. Below is a list of supported models of each runtime and its corresponding size. We include the disk usage (in delta) and the peak RAM and VRAM usage (in delta) when running on a single Nvidia TITAN RTX GPU (24GB VRAM) for a series of text and image encoding tasks with `batch_size=8` using PyTorch runtime.

| Model | PyTorch | ONNX | TensorRT | Output Dimension | Disk Usage (MB) | Peak RAM Usage (GB) | Peak VRAM Usage (GB) |
|---------------------------------------|---------|------|----------|------------------|-----------------|---------------------|----------------------|
| RN50 | ✅ | ✅ | ✅ | 1024 | 244 | 2.99 | 1.36 |
| RN101 | ✅ | ✅ | ✅ | 512 | 278 | 3.05 | 1.40 |
| RN50x4 | ✅ | ✅ | ✅ | 640 | 402 | 3.23 | 1.63 |
| RN50x16 | ✅ | ✅ | ❌ | 768 | 631 | 3.63 | 2.02 |
| RN50x64 | ✅ | ✅ | ❌ | 1024 | 1291 | 4.08 | 2.98 |
| ViT-B-32 | ✅ | ✅ | ✅ | 512 | 338 | 3.20 | 1.40 |
| ViT-B-16 | ✅ | ✅ | ✅ | 512 | 335 | 3.20 | 1.44 |
| ViT-B-16-plus-240 | ✅ | ✅ | 🚧 | 640 | 795 | 3.03 | 1.59 |
| ViT-L-14 | ✅ | ✅ | ❌ | 768 | 890 | 3.66 | 2.04 |
| ViT-L-14@336px | ✅ | ✅ | ❌ | 768 | 891 | 3.74 | 2.23 |
| M-CLIP/XLM-Roberta-Large-Vit-B-32 | ✅ | 🚧 | 🚧 | 512 | 4284 | 5.37 | 1.68 |
| M-CLIP/XLM-Roberta-Large-Vit-L-14 | ✅ | 🚧 | ❌ | 768 | 4293 | 4.30 | 4.97 |
| M-CLIP/XLM-Roberta-Large-Vit-B-16Plus | ✅ | 🚧 | 🚧 | 640 | 4293 | 4.30 | 4.13 |
| M-CLIP/LABSE-Vit-L-14 | ✅ | 🚧 | ❌ | 768 | 3609 | 4.30 | 4.70 |

✅ = First class support — 🚧 = Unsupported, working in progress

`ViT-B-32::openai` is used as the default model in all runtimes. To use specific pretrained models provided by `open_clip`, please use `::` to separate model name and pretrained weight name, e.g. `ViT-B-32::laion2b_e16`.
Full list of open_clip models and weights can be found [here](https://github.com/mlfoundations/open_clip#pretrained-model-interface).

```{note}
For model definition with `-quickgelu` postfix, please use non `-quickgelu` model name.
```

### Use custom model for onnx
You can also use your own model in ONNX runtime by specifying the model name and the path to ONNX model directory in YAML file.
The model directory should have the same structure as below:

```text
Expand Down Expand Up @@ -249,7 +264,7 @@ For all backends, you can set the following parameters via `with`:

| Parameter | Description |
|-------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| `name` | Model weights, default is `ViT-B/32`. Support all OpenAI released pretrained models. |
| `name` | Model weights, default is `ViT-B-32::openai`. A full list of models and weights can be found [here](#model-support) |
| `num_worker_preprocess` | The number of CPU workers for image & text prerpocessing, default 4. |
| `minibatch_size` | The size of a minibatch for CPU preprocessing and GPU encoding, default 64. Reduce the size of it if you encounter OOM on GPU. |

Expand Down

0 comments on commit c67a7f5

Please sign in to comment.