cannot get_mask when I vary the cuda device

[ada-cheng]

Hello Junyi, GREAT JOB! It seems that everything works well when calling get_features in extractor_sd.py using cuda:3
but the inference process failed even I change
def inference(model, aug, image, vocab, label_list):
from
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)

pred = demo.predict(np.array(image))
to
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)

demo.model = demo.model.to(torch.device("cuda:3"))

pred = demo.predict(np.array(image))

I guess the main problem lies in wrongly loading the decoder part of the model, but I'm not sure how to fix it.

[ada-cheng]

also I guess I loaded the model to the expected cuda
model, aug = load_model(diffusion_ver=VER, image_size=SIZE, num_timesteps=TIMESTEP, block_indices=tuple(INDICES), decoder_only=False)
model = model.to(device)
since I have added these lines in the pck_spair_pascal file.

[Junyi42]

Hi, thanks for the interest in our work!

Interestingly, I met a similar problem when I first played with the ODISE repo😂. And here's the issue I raised in their repo. I ended up getting around this problem by setting the CUDA_VISIBLE_DEVICES instead of the torch.to(device) (e.g., you can add another line os.environ["CUDA_VISIBLE_DEVICES"]=3 on top of the script). Hopefully it could be helpful for you.

Best,
Junyi

[ada-cheng]

Great! I've successfully obtained the rainbow mask. I'm truly grateful for your assistance.