Runtime Error Using DirectML Device

[FXJLK]

Using the train script for the yolov3 sample I get this error when trying to use DML as a device:

[libprotobuf FATAL D:\a_work\1\s\pytorch-directml\caffe2\dml\dml_operator.cc:116] CHECK failed: ((((HRESULT)((backend_->dml_device->CreateOperator(&op_desc_, uuidof(**(&op)), IID_PPV_ARGS_Helper(&op))))) >= 0)) == (true):
Traceback (most recent call last):
File "train.py", line 543, in
train(hyp, opt, device, tb_writer)
File "train.py", line 235, in train
model.class_weights = labels_to_class_weights(dataset.labels, nc).to(device) * nc # attach class weights
RuntimeError: CHECK failed: ((((HRESULT)((backend->dml_device->CreateOperator(&op_desc, __uuidof(**(&op)), IID_PPV_ARGS_Helper(&op))))) >= 0)) == (true):

I can use detect.py with both CPU and DML and I can also use train.py using the cpu but I get this wierd error trying to train using DML.

[zhangxiang1993]

Hi @FXJLK, Can you print out the dtype and shape of the returned tensor from labels_to_class_weights(dataset.labels, nc). It looks nothing is involved with pytorch-directml before .to(device).

[FXJLK]

Thank you for the reply. The data type is torch.float64 and the shape returns torch.Size([80]).

[FXJLK]

@zhangxiang1993 Could it be a compatibility issue with my GPU? I am using a RX 5700 with the latest drivers on windows 10 on build 19044.

It seems like the dml device can do the pytorch operations but whenever the tensor on the dml device is used in certain ways that same error appears. Code example:

import torch
x = torch.ones(5, device='dml')
y = x + x
print(y)

[libprotobuf FATAL D:\a_work\1\s\pytorch-directml\caffe2\dml\dml_operator.cc:116] CHECK failed: ((((HRESULT)((backend_->dml_device->CreateOperator(&op_desc_, __uuidof((&op)), IID_PPV_ARGS_Helper(&op))))) >= 0)) == (true):
Traceback (most recent call last):
File "", line 1, in
File "E:\Software2\miniconda\envs\yolo\lib\site-packages\torch\tensor.py", line 193, in repr
return torch._tensor_str._str(self)
File "E:\Software2\miniconda\envs\yolo\lib\site-packages\torch_tensor_str.py", line 383, in _str
return _str_intern(self)
File "E:\Software2\miniconda\envs\yolo\lib\site-packages\torch_tensor_str.py", line 358, in _str_intern
tensor_str = _tensor_str(self, indent)
File "E:\Software2\miniconda\envs\yolo\lib\site-packages\torch_tensor_str.py", line 242, in tensor_str
formatter = Formatter(get_summarized_data(self) if summarize else self)
File "E:\Software2\miniconda\envs\yolo\lib\site-packages\torch_tensor_str.py", line 109, in init
if nonzero_finite_max / nonzero_finite_min > 1000. or nonzero_finite_max > 1.e8:
RuntimeError: CHECK failed: ((((HRESULT)((backend->dml_device->CreateOperator(&op_desc, __uuidof((&op)), IID_PPV_ARGS_Helper(&op))))) >= 0)) == (true):

y = y.to('cpu')
print(y)

tensor([2., 2., 2., 2., 2.])

[linnealovespie]

Hi @FXJLK, your latest error is due to printing a floating-point DML tensor. This is currently not supported and an explicit cast to CPU is required. We will address this in a future release.

Float64 support is still sparse in torch-directml. The latest release of torch-directml-0.1.13.dev221216 and samples should avoid using float64 by downcasting to float32. Please give it a shot and see if this works for you.

If you're still experiencing issues, you may be running out of memory, per #306. The latest version should give you a friendly OOM error message to help debug.