RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

[sushovanjena]

Traceback (most recent call last):
File "/home/arnav/Sushovan/yolov7-main/train.py", line 622, in
train(hyp, opt, device, tb_writer)
File "/home/arnav/Sushovan/yolov7-main/train.py", line 369, in train
loss, loss_items = compute_loss_ota(pred, targets, imgs) # loss scaled by batch_size changed
File "/home/arnav/Sushovan/yolov7-main/utils/loss.py", line 585, in call
bs, as_, gjs, gis, targets, anchors = self.build_targets(p, targets, imgs)
File "/home/arnav/Sushovan/yolov7-main/utils/loss.py", line 759, in build_targets
from_which_layer = from_which_layer[fg_mask_inboxes]
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Please help in solving the error. Do I need to transfer some part of code to gpu ? Then which part ?
Actually, it was working fine in simple GPU, but showing this error in HPC.

[jeffacce]

https://github.com/WongKinYiu/yolov7/blob/main/utils/loss.py#L742

Changing this line to

matching_matrix = torch.zeros_like(cost, device="cpu")

worked for me.

[sushovanjena]

Bro, it worked for me. Your help is godly. I have been trying to solve this for 4 days. Lots of thanks.
But I don't understand, when the same code was able to run properly on GPU, what problem is it getting on HPC.

[sushovanjena]

I am wandering, even if we dont mention device="cpu" explicitly, its gets created in cpu only right ?
why to mention "CPU" explicitly ?

[jeffacce]

I think torch.zeros_like(x) by default allocates to the same device as x, which is on GPU in this case.

[rakshith-ramagiri]

If you're training P6 models like e6 or w6 or x, then you'll need to change the following lines as well:

• 1389 - matching_matrix = torch.zeros_like(cost) to matching_matrix = torch.zeros_like(cost, device="cpu")
• 1543 - matching_matrix = torch.zeros_like(cost) to matching_matrix = torch.zeros_like(cost, device="cpu")

in the same file (utils/loss.py).

[alexandrerays]

Great, that worked for me!

[Manpreetkour95]

This is occurring due to device issue. I run the same code on colab it worked perfectly fine. But when I tried using AWS it was giving this error. The above answer worked for me. Thanks.

[Boualytpv]

https://github.com/WongKinYiu/yolov7/blob/main/utils/loss.py#L742

Changing this line to

matching_matrix = torch.zeros_like(cost, device="cpu")

worked for me.

[RANA-ATI]

matching_matrix = torch.zeros_like(cost, device="cuda") at line 742 in loss.py worked for me

[etale-cohomology]

matching_matrix = torch.zeros_like(cost, device="cpu")

wasn't working, but

matching_matrix = torch.zeros_like(cost, device="cuda")

did, probably because I had already modified a couple lines in loss.py to set the device to cuda.

[HUAYEFE]

如果您正在训练P6像这样的模型e6 or w6 or x，那么您还需要更改以下行：

• 1389-matching_matrix = torch.zeros_like(cost)到matching_matrix = torch.zeros_like(cost, device="cpu")
• 1543-matching_matrix = torch.zeros_like(cost)到matching_matrix = torch.zeros_like(cost, device="cpu")

在同一文件中 ( utils/loss.py)。

I spent 3-4 hours trying other methods, and in the end, I successfully ran YOLO7 training w6 on August 6, 2023. Thank you

[LeAyky]

如果您正在训练P6像这样的模型e6 or w6 or x，那么您还需要更改以下行：

• 1389-matching_matrix = torch.zeros_like(cost)到matching_matrix = torch.zeros_like(cost, device="cpu")
• 1543-matching_matrix = torch.zeros_like(cost)到matching_matrix = torch.zeros_like(cost, device="cpu")

在同一文件中 ( utils/loss.py)。

I spent 3-4 hours trying other methods, and in the end, I successfully ran YOLO7 training w6 on August 6, 2023. Thank you

I feel you. Would really appreciate, if it would get fixed. Thank you! :) @WongKinYiu