Error executing DirectML PyTorch tutorial #287
Description
I followed the exactly same steps of the DirectML turorial and got the following error trying to run the Pytorch squeezenet example:
RuntimeError: CHECK failed: ((((HRESULT)((dml_device_->GetDeviceRemovedReason()))) >= 0)) == (true):
Full stdout and error stack:
Loading the training dataset from: C:\Users\user\Documents\DirectML\PyTorch\data\cifar-10-python
Train data X [N, C, H, W]:
shape=torch.Size([32, 3, 224, 224]),
dtype=torch.float32
Train data Y:
shape=torch.Size([32]),
dtype=torch.int64
Loading the testing dataset from: C:\Users\user\Documents\DirectML\PyTorch\data\cifar-10-python
Test data X [N, C, H, W]:
shape=torch.Size([32, 3, 224, 224]),
dtype=torch.float32
Test data Y:
shape=torch.Size([32]),
dtype=torch.int64
Finished moving squeezenet1_1 to device: dml in 0.0s.
Epoch 1
[libprotobuf FATAL D:\a_work\1\s\pytorch-directml\caffe2\dml\dml_command_recorder.cc:392] CHECK failed: ((((HRESULT)((dml_device_->GetDeviceRemovedReason()))) >= 0)) == (true):
Traceback (most recent call last):
File ".\pytorch\squeezenet\train.py", line 35, in
main()
File ".\pytorch\squeezenet\train.py", line 30, in main
train(args.path, args.batch_size, args.epochs, args.learning_rate,
File "C:\Users\user\Documents\DirectML\PyTorch\classification\train_classification.py", line 133, in main
train(training_dataloader,
File "C:\Users\user\Documents\DirectML\PyTorch\classification\train_classification.py", line 87, in train
optimizer.step()
File "C:\Users\user\miniconda3\envs\pydml\lib\site-packages\torch\optim\optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "C:\Users\user\miniconda3\envs\pydml\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\user\miniconda3\envs\pydml\lib\site-packages\torch\optim\sgd.py", line 110, in step
F.sgd(params_with_grad,
File "C:\Users\user\miniconda3\envs\pydml\lib\site-packages\torch\optim_functional.py", line 169, in sgd
buf.mul_(momentum).add_(d_p, alpha=1 - dampening)
RuntimeError: CHECK failed: ((((HRESULT)((dml_device_->GetDeviceRemovedReason()))) >= 0)) == (true):
My machine specs:
OS Name: Microsoft Windows 11 Pro
OS Version: 10.0.22000 N/A Build 22000
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Original Install Date: 8/26/2022, 1:56:03 PM
System Boot Time: 8/27/2022, 11:32:38 PM
System Manufacturer: Dell Inc.
System Model: Latitude 5530
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 154 Stepping 4 GenuineIntel ~1700 Mhz
BIOS Version: Dell Inc. 1.5.0, 7/8/2022
System Locale: en-us;English (United States)
Input Locale: es-mx;Spanish (Mexico)
Time Zone: (UTC-06:00) Guadalajara, Mexico City, Monterrey
Total Physical Memory: 16,069 MB
Available Physical Memory: 2,215 MB
Virtual Memory: Max Size: 27,669 MB
Virtual Memory: Available: 5,720 MB
Virtual Memory: In Use: 21,949 MB
Page File Location(s): C:\pagefile.sys
Domain: WORKGROUP
Logon Server: \DESKTOP-EOH8KF9
Hotfix(s): 6 Hotfix(s) Installed.
[01]: KB5015732
[02]: KB5007575
[03]: KB5012170
[04]: KB5016629
[05]: KB5006755
[06]: KB5015898
Network Card(s): 2 NIC(s) Installed.
[01]: Intel(R) Wi-Fi 6E AX211 160MHz
Connection Name: Wi-Fi
DHCP Enabled: Yes
[02]: Intel(R) Ethernet Connection (16) I219-LM
Connection Name: Ethernet
Status: Media disconnected
Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.
Python 3.8
Package Version
absl-py 1.2.0
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.0.8
attrs 22.1.0
backcall 0.2.0
beautifulsoup4 4.11.1
bleach 5.0.1
cachetools 5.2.0
certifi 2022.6.15
cffi 1.15.1
charset-normalizer 2.1.1
colorama 0.4.5
cycler 0.11.0
debugpy 1.6.3
decorator 5.1.1
defusedxml 0.7.1
entrypoints 0.4
executing 1.0.0
fastjsonschema 2.16.1
fonttools 4.37.1
google-auth 2.11.0
google-auth-oauthlib 0.4.6
grpcio 1.48.1
idna 3.3
importlib-metadata 4.12.0
importlib-resources 5.9.0
ipykernel 6.15.2
ipython 8.4.0
ipython-genutils 0.2.0
ipywidgets 8.0.1
jedi 0.18.1
Jinja2 3.1.2
jsonschema 4.15.0
jupyter 1.0.0
jupyter_client 7.3.5
jupyter-console 6.4.4
jupyter-core 4.11.1
jupyterlab-pygments 0.2.2
jupyterlab-widgets 3.0.2
kiwisolver 1.4.4
lxml 4.9.1
Markdown 3.4.1
MarkupSafe 2.1.1
matplotlib 3.5.3
matplotlib-inline 0.1.6
mistune 2.0.4
nbclient 0.6.7
nbconvert 7.0.0
nbformat 5.4.0
nest-asyncio 1.5.5
notebook 6.4.12
numpy 1.23.2
oauthlib 3.2.0
opencv-python 4.6.0.66
packaging 21.3
pandas 1.4.4
pandocfilters 1.5.0
parso 0.8.3
pickleshare 0.7.5
Pillow 9.2.0
pip 22.1.2
pkgutil_resolve_name 1.3.10
prometheus-client 0.14.1
prompt-toolkit 3.0.31
protobuf 3.19.4
psutil 5.9.1
pure-eval 0.2.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
Pygments 2.13.0
pyparsing 3.0.9
pyrsistent 0.18.1
python-dateutil 2.8.2
pytorch-directml 1.8.0a0.dev220506
pytz 2022.2.1
pywin32 304
pywinpty 2.0.7
PyYAML 6.0
pyzmq 23.2.1
qtconsole 5.3.2
QtPy 2.2.0
requests 2.28.1
requests-oauthlib 1.3.1
rsa 4.9
Send2Trash 1.8.0
setuptools 63.4.1
six 1.16.0
soupsieve 2.3.2.post1
stack-data 0.5.0
tensorboard 2.10.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
terminado 0.15.0
tinycss2 1.1.1
torchvision 0.9.0
tornado 6.2
tqdm 4.64.0
traitlets 5.3.0
typing_extensions 4.3.0
urllib3 1.26.12
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 2.2.2
wget 3.2
wheel 0.37.1
widgetsnbextension 4.0.2
wincertstore 0.2
zipp 3.8.1