RuntimeError: CUDA error: unspecified launch failure

[dgiunchi]

Hi I have frequently this error. I reduced the inputs but it appears very often without predicibility.

2300 46.0% 129m 3s content_loss:66.045113 style_loss:150.286469 total_loss:216.331573
Traceback (most recent call last):
File "train.py", line 100, in
content_loss.item(),
RuntimeError: CUDA error: unspecified launch failure

my conda environment:
absl-py @ file:///D:/bld/absl-py_1598382479526/work
astor @ file:///home/conda/feedstock_root/build_artifacts/astor_1593610464257/work
audioread==2.1.8
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache==1.6.1
certifi==2020.6.20
cffi @ file:///D:/bld/cffi_1595805721274/work
colorama==0.4.3
colorlog==4.2.1
cycler==0.10.0
decorator==4.4.2
future==0.18.2
gast @ file:///home/conda/feedstock_root/build_artifacts/gast_1596839682936/work
google-pasta==0.2.0
grpcio @ file:///D:/bld/grpcio_1596715850903/work
h5py @ file:///D:/bld/h5py_1595110299148/work
hydra-colorlog==0.1.4
hydra-core==0.11.3
importlib-metadata @ file:///D:/bld/importlib-metadata_1593211612489/work
inflect==0.2.5
ipykernel @ file:///D:/bld/ipykernel_1595447157738/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython @ file:///D:/bld/ipython_1598750239682/work
ipython-genutils==0.2.0
jedi @ file:///D:/bld/jedi_1595018503144/work
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1593624380152/work
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1598486169312/work
jupyter-core==4.6.3
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.2.0
librosa==0.6.3
llvmlite==0.31.0
Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1589366472132/work
matplotlib==2.2.5
mkl-service==2.3.0
numba==0.48.0
numpy @ file:///D:/bld/numpy_1597938567428/work
olefile==0.46
omegaconf==1.4.1
opt-einsum==3.3.0
pandas==1.1.1
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1595548966091/work
pickleshare==0.7.5
Pillow @ file:///D:/bld/pillow_1594213048891/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1598885455507/work
protobuf==3.13.0
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
Pygments==2.6.1
pyparsing==2.4.7
PyQt5==5.12.3
PyQt5-sip==4.19.18
PyQtWebEngine==5.12.1
pyreadline==2.1
pystoi==0.3.3
python-dateutil==2.8.1
pytz==2020.1
pywin32==227
PyYAML==5.3.1
pyzmq==19.0.2
resampy==0.2.2
scikit-learn @ file:///D:/bld/scikit-learn_1596546337481/work
scipy @ file:///C:/ci/scipy_1597686737426/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
sounddevice==0.4.0
tensorboard==1.15.0
tensorboardX @ file:///home/conda/feedstock_root/build_artifacts/tensorboardx_1594067496847/work
tensorflow-estimator==1.15.1
tensorflow-gpu==1.15.2
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp79xdzxkt/threadpoolctl-2.1.0-py3-none-any.whl
torch==1.6.0
torchaudio==0.6.0
torchvision==0.7.0
tornado==6.0.4
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1596476591553/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1599170348041/work
Unidecode==1.0.22
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1595859607677/work
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1
zipp==3.1.0

[mazzzystar]

Hi, see if pytorch/pytorch#27837 can help you~