Unable to run on GPU with error: "Cannot copy out of meta tensor; no data!"

[shomikj]

I'm trying to follow the instructions in #228 in order to run inference on a GPU.

My code looks as follows:

model, image_processor, tokenizer = create_model_and_transforms(
    clip_vision_encoder_path="ViT-L-14",
    clip_vision_encoder_pretrained="openai",
    lang_encoder_path="anas-awadalla/mpt-1b-redpajama-200b",
    tokenizer_path="anas-awadalla/mpt-1b-redpajama-200b",
    cross_attn_every_n_layers=1
)

checkpoint_path = hf_hub_download("openflamingo/OpenFlamingo-3B-vitl-mpt1b", "checkpoint.pt")
model.load_state_dict(torch.load(checkpoint_path, map_location="cuda:0"), strict=False)

device = torch.device("cuda:0")
model.to(device)

In order to avoid the multi-GPU problem specified in #243, I am running my script with the flag CUDA_VISIBLE_DEVICES=0 python test_flamingo.py

However, I am still getting a similar error on the model.to(device) step:

Flamingo model initialized with 1046992944 trainable parameters
Traceback (most recent call last):
  File "/u/shomikj/ring/open_flamingo-main/test_flamingo.py", line 35, in <module>
    model.to(device)
  File "/u/shomikj/miniconda3/envs/openflamingo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "/u/shomikj/miniconda3/envs/openflamingo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/u/shomikj/miniconda3/envs/openflamingo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/u/shomikj/miniconda3/envs/openflamingo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/u/shomikj/miniconda3/envs/openflamingo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 820, in _apply
    param_applied = fn(param)
  File "/u/shomikj/miniconda3/envs/openflamingo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

[shomikj]

@anas-awadalla would you have any ideas for this? thank you!

[QimingLee]

I just encountered the same issue, and you can resolve it by modifying the device information from "meta" to "cuda" within the JSON file, allowing the program to run properly.