[bug]If the 'gpu_id' parameter of minigptv2 is not 0, an error will occur

[lckj2009]

First of all, I would like to express my gratitude to "Deyao Zhu *, Jun Chen *, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny" and others for their hard work, which has led to the current minigptv2.

It has been confirmed that in the current version of minigptv2, if the "-- gpu id" parameter is 0, there will be no error reported. If it's not 0, but 1. The following errors may occur when answering:

Traceback (most recent call last):
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/queueing.py", line 406, in call_prediction
output = await route_utils.call_process_api(
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/route_utils.py", line 226, in call_process_api
output = await app.get_blocks().process_api(
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/blocks.py", line 1555, in process_api
result = await self.call_function(
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/blocks.py", line 1207, in call_function
prediction = await utils.async_iteration(iterator)
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/utils.py", line 517, in async_iteration
return await iterator.anext()
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/utils.py", line 510, in anext
return await anyio.to_thread.run_sync(
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/utils.py", line 493, in run_sync_iterator_async
return next(iterator)
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/gradio/utils.py", line 647, in gen_wrapper
yield from f(*args, **kwargs)
File "/root/MiniGPT-4/demo_v2.py", line 468, in gradio_stream_answer
streamer = chat.stream_answer(conv=chat_state,
File "/root/MiniGPT-4/minigpt4/conversation/conversation.py", line 197, in stream_answer
generation_kwargs = self.answer_prepare(conv, img_list, **kargs)
File "/root/MiniGPT-4/minigpt4/conversation/conversation.py", line 162, in answer_prepare
embs = self.model.get_context_emb(prompt, img_list)
File "/root/MiniGPT-4/minigpt4/models/minigpt_base.py", line 77, in get_context_emb
seg_embs = [self.embed_tokens(seg_t) for seg_t in seg_tokens]
File "/root/MiniGPT-4/minigpt4/models/minigpt_base.py", line 77, in
seg_embs = [self.embed_tokens(seg_t) for seg_t in seg_tokens]
File "/root/MiniGPT-4/minigpt4/models/minigpt_base.py", line 312, in embed_tokens
embeds = self.llama_model.base_model.model.model.embed_tokens(token_ids)
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
result = forward_call(*args, **kwargs)
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 162, in forward
return F.embedding(
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/torch/nn/functional.py", line 2210, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

Although we can try to modify this issue ourselves, we still hope that your team can solve this bug.

[Oklahomawhore]

There is a bug in minigptv2 initialization, the config are not correctly passed into init method.
add device_8bit = cfg.get("device_8bit", 0) to minigpt_v2.py line 102

[lckj2009]

device_8bit = cfg.get("device_8bit", 0)

Hello, the addition of 'device_8bit = cfg.get("device_8bit", 0)' is still the same error. Do we need to change something else as well?

My startup command is : python demo_v2.py --cfg-path eval_configs/minigptv2_eval.yaml --gpu-id 1

File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 162, in forward
return F.embedding(
File "/root/anaconda3/envs/minigptv/lib/python3.9/site-packages/torch/nn/functional.py", line 2216, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

[Oklahomawhore]

check #439

[lckj2009]

check #439

thank you.Resolved. I added one less line of code later on