[Q] Minimum GPU Ram requirements for inference on models?

[NightMachinery]

What minimum GPU ram do I need for inference on each model, e.g., ViT-G/14?

I couldn't find 8-bit inference guides using pretrained weights. Is this possible? What would be the RAM requirements then?

[mitchellnw]

I don't think we know the exact amount of GPU RAM required, but you can follow this tutorial for 8-bit inference https://github.com/mlfoundations/open_clip/blob/main/tutorials/int8_tutorial.ipynb

Concretely, your going to want to do something like the following (note I didn't actually check this code, just copied from the tutorial so there may be a typo, please double check tutorial if it doesn't work)

import open_clip
import bitsandbytes as bnb

model, _, preprocess = open_clip.create_model_and_transforms('ViT-H-14', pretrained='laion2b_s32b_b79k')
int8_linear_layer = bnb.nn.triton_based_modules.SwitchBackLinear
int8_model = open_clip.utils.replace_linear(model, int8_linear_layer, include_modules=['c_fc', 'c_proj']).cuda()
int8_model.eval()

from open_clip.utils import convert_int8_model_to_inference_mode
convert_int8_model_to_inference_mode(int8_model)

# Now you can do a forward pass with the model...

[mobassir94]

hi @mitchellnw
with your above mentioned code in kaggle i face this error

=> preparing for eval.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_23/1251702524.py in <module>
      9 
     10 from open_clip.utils import convert_int8_model_to_inference_mode
---> 11 convert_int8_model_to_inference_mode(int8_model)
     12 
     13 # Now you can do a forward pass with the model...

/opt/conda/lib/python3.7/site-packages/open_clip/utils.py in convert_int8_model_to_inference_mode(model)
     86         if hasattr(m, 'prepare_for_eval'):
     87             int8_original_dtype = m.weight.dtype
---> 88             m.prepare_for_eval()
     89             m.int8_original_dtype = int8_original_dtype

/opt/conda/lib/python3.7/site-packages/bitsandbytes/nn/triton_based_modules.py in prepare_for_eval(self)
    192             W_int8, state_W = quantize_rowwise(self.weight)
    193         else:
--> 194             W_int8, state_W = quantize_global(self.weight)
    195 
    196         self.register_buffer("W_int8", W_int8)

TypeError: cannot unpack non-iterable NoneType object

[mitchellnw]

It looks like this is happening because triton is not installed

https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/triton/quantize_global.py#L6-L8

if you install triton it should fix it

[mobassir94]

@mitchellnw i have cudatoolkit 11.7.0
i install triton and bnb like this

!pip install triton==2.0.0.post1
!pip install bitsandbytes-cuda117 -q

and i face no error during installation,

Collecting triton==2.0.0.post1
  Downloading triton-2.0.0.post1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (63.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 MB 19.3 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: cmake in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0.post1) (3.25.0)
Collecting lit
  Downloading lit-16.0.2.tar.gz (137 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.9/137.9 kB 12.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0.post1) (1.13.0)
Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0.post1) (3.9.0)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0.post1) (4.4.0)
Building wheels for collected packages: lit
  Building wheel for lit (setup.py) ... done
  Created wheel for lit: filename=lit-16.0.2-py3-none-any.whl size=88191 sha256=8b97f91564420a395997a1e7065f9aca41dfa1719e2f79ca8614642d072d8c9f
  Stored in directory: /root/.cache/pip/wheels/07/98/43/4588455ecb4d600b42aae1591fc9895fac682c07fc1213ac3c
Successfully built lit
Installing collected packages: lit, triton
Successfully installed lit-16.0.2 triton-2.0.0.post1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: [https://pip.pypa.io/warnings/venv](https://pip.pypa.io/warnings/venv%3C/span%3E%3Cspan) class="ansi-yellow-fg">
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: [https://pip.pypa.io/warnings/venv class="ansi-yellow-fg">](https://pip.pypa.io/warnings/venv%3C/span%3E%3Cspan)

and then when i run this code

import bitsandbytes as bnb

from bitsandbytes.triton.triton_utils import is_triton_available
is_triton_available()

i face this error

==============================WARNING: DEPRECATED!==============================
WARNING! This version of bitsandbytes is deprecated. Please switch to `pip install bitsandbytes` and the new repo: https://github.com/TimDettmers/bitsandbytes
==============================WARNING: DEPRECATED!==============================
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/tmp/ipykernel_23/1480283209.py in <module>
      1 import bitsandbytes as bnb
      2 
----> 3 from bitsandbytes.triton.triton_utils import is_triton_available
      4 is_triton_available()

ModuleNotFoundError: No module named 'bitsandbytes.triton'

any idea what mistake i am making?

[mitchellnw]

The error is probably related to:

WARNING! This version of bitsandbytes is deprecated. Please switch to `pip install bitsandbytes` and the new repo: https://github.com/TimDettmers/bitsandbytes

So seems something is wrong with your bitsandbytes instillation. I would recommend filing an issue there: https://github.com/TimDettmers/bitsandbytes