We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Am suffering from CUDA Error during I was running Trackformer on VisDrone2019 dataset on a server with 3 RTX3090 GPUs:
Start training /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [34,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [5,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [1,0,0], thread: [27,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. Traceback (most recent call last): File "src/train.py", line 357, in train(args) File "src/train.py", line 284, in train visualizers['train'], args) File "/root/autodl-tmp/trackformer/src/trackformer/engine.py", line 128, in train_one_epoch loss_dict = criterion(outputs, targets) File "/root/miniconda3/envs/tf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 406, in forward losses.update(self.get_loss(loss, outputs, targets, indices, num_boxes)) File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 380, in get_loss return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs) File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 252, in loss_labels_focal losses['class_error'] = 100 - accuracy(src_logits[idx], target_classes_o)[0] RuntimeError: CUDA error: device-side assert triggered
My conda environment is listed as below:
_libgcc_mutex 0.1 main https://mirrors.ustc.edu.cn/anaconda/pkgs/main _openmp_mutex 5.1 1_gnu https://mirrors.ustc.edu.cn/anaconda/pkgs/main argon2-cffi 20.1.0 pypi_0 pypi astroid 2.4.2 pypi_0 pypi async-generator 1.10 pypi_0 pypi attrs 19.3.0 pypi_0 pypi backcall 0.2.0 pypi_0 pypi bleach 3.2.3 pypi_0 pypi ca-certificates 2023.5.7 hbcca054_0 conda-forge certifi 2020.4.5.2 pypi_0 pypi cffi 1.14.4 pypi_0 pypi chardet 3.0.4 pypi_0 pypi cloudpickle 1.6.0 pypi_0 pypi colorama 0.4.3 pypi_0 pypi cycler 0.10.0 pypi_0 pypi cython 0.29.20 pypi_0 pypi decorator 4.4.2 pypi_0 pypi defusedxml 0.6.0 pypi_0 pypi docopt 0.6.2 pypi_0 pypi entrypoints 0.3 pypi_0 pypi filelock 3.0.12 pypi_0 pypi flake8 3.8.3 pypi_0 pypi flake8-import-order 0.18.1 pypi_0 pypi future 0.18.2 pypi_0 pypi gdown 3.12.2 pypi_0 pypi gitdb 4.0.5 pypi_0 pypi gitpython 3.1.3 pypi_0 pypi idna 2.9 pypi_0 pypi imageio 2.8.0 pypi_0 pypi importlib-metadata 1.6.1 pypi_0 pypi ipykernel 5.4.3 pypi_0 pypi ipython 7.19.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 7.6.3 pypi_0 pypi isort 5.6.4 pypi_0 pypi jedi 0.18.0 pypi_0 pypi jinja2 2.11.2 pypi_0 pypi jsonpatch 1.25 pypi_0 pypi jsonpickle 1.4.1 pypi_0 pypi jsonpointer 2.0 pypi_0 pypi jsonschema 3.2.0 pypi_0 pypi jupyter 1.0.0 pypi_0 pypi jupyter-client 6.1.11 pypi_0 pypi jupyter-console 6.2.0 pypi_0 pypi jupyter-core 4.7.0 pypi_0 pypi jupyterlab-pygments 0.1.2 pypi_0 pypi jupyterlab-widgets 1.0.0 pypi_0 pypi kiwisolver 1.2.0 pypi_0 pypi lap 0.4.0 py37hfb7772e_1005 conda-forge lapsolver 1.1.0 pypi_0 pypi lazy-object-proxy 1.4.3 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main libblas 3.9.0 15_linux64_openblas conda-forge libcblas 3.9.0 15_linux64_openblas conda-forge libffi 3.4.4 h6a678d5_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main libgcc-ng 11.2.0 h1234567_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main libgfortran-ng 13.1.0 h69a702a_0 conda-forge libgfortran5 13.1.0 h15d22d2_0 conda-forge libgomp 11.2.0 h1234567_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main liblapack 3.9.0 15_linux64_openblas conda-forge libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge libstdcxx-ng 11.2.0 h1234567_1 https://mirrors.ustc.edu.cn/anaconda/pkgs/main markupsafe 1.1.1 pypi_0 pypi matplotlib 3.2.1 pypi_0 pypi mccabe 0.6.1 pypi_0 pypi mistune 0.8.4 pypi_0 pypi more-itertools 8.4.0 pypi_0 pypi motmetrics 1.2.0 pypi_0 pypi multiscaledeformableattention 1.0 pypi_0 pypi munch 2.5.0 pypi_0 pypi nbclient 0.5.1 pypi_0 pypi nbconvert 6.0.7 pypi_0 pypi nbformat 5.1.2 pypi_0 pypi ncurses 6.4 h6a678d5_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main nest-asyncio 1.5.1 pypi_0 pypi networkx 2.4 pypi_0 pypi ninja 1.10.0.post2 pypi_0 pypi notebook 6.2.0 pypi_0 pypi numpy 1.18.5 pypi_0 pypi opencv-python 4.2.0.34 pypi_0 pypi openssl 1.1.1u h7f8727e_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main packaging 20.4 pypi_0 pypi pandas 1.0.5 pypi_0 pypi pandocfilters 1.4.3 pypi_0 pypi parso 0.8.1 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 7.1.2 pypi_0 pypi pip 22.3.1 py37h06a4308_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main pluggy 0.13.1 pypi_0 pypi prometheus-client 0.9.0 pypi_0 pypi prompt-toolkit 3.0.14 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi py 1.8.2 pypi_0 pypi py-cpuinfo 6.0.0 pypi_0 pypi pyaml 20.4.0 pypi_0 pypi pycocotools 2.0 pypi_0 pypi pycodestyle 2.6.0 pypi_0 pypi pycparser 2.20 pypi_0 pypi pyflakes 2.2.0 pypi_0 pypi pygments 2.7.4 pypi_0 pypi pylint 2.6.0 pypi_0 pypi pyparsing 2.4.7 pypi_0 pypi pyrsistent 0.17.3 pypi_0 pypi pysocks 1.7.1 pypi_0 pypi pytest 5.4.3 pypi_0 pypi pytest-benchmark 3.2.3 pypi_0 pypi python 3.7.16 h7a1cb2a_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main python-dateutil 2.8.1 pypi_0 pypi python_abi 3.7 2_cp37m conda-forge pytz 2020.1 pypi_0 pypi pywavelets 1.1.1 pypi_0 pypi pyyaml 5.3.1 pypi_0 pypi pyzmq 19.0.1 pypi_0 pypi qtconsole 5.0.2 pypi_0 pypi qtpy 1.9.0 pypi_0 pypi readline 8.2 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main requests 2.23.0 pypi_0 pypi sacred 0.8.1 pypi_0 pypi scikit-image 0.17.2 pypi_0 pypi scipy 1.4.1 pypi_0 pypi seaborn 0.10.1 pypi_0 pypi send2trash 1.5.0 pypi_0 pypi setuptools 65.6.3 py37h06a4308_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main six 1.15.0 pypi_0 pypi smmap 3.0.4 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main submitit 1.1.5 pypi_0 pypi terminado 0.9.2 pypi_0 pypi testpath 0.4.4 pypi_0 pypi tifffile 2020.6.3 pypi_0 pypi tk 8.6.12 h1ccaba5_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main toml 0.10.2 pypi_0 pypi torch 1.8.1+cu111 pypi_0 pypi torchaudio 0.8.1 pypi_0 pypi torchfile 0.1.0 pypi_0 pypi torchvision 0.9.1+cu111 pypi_0 pypi tornado 6.1 pypi_0 pypi tqdm 4.46.1 pypi_0 pypi traitlets 5.0.5 pypi_0 pypi typed-ast 1.4.1 pypi_0 pypi typing-extensions 3.7.4.3 pypi_0 pypi urllib3 1.25.9 pypi_0 pypi visdom 0.1.8.9 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 0.57.0 pypi_0 pypi wheel 0.38.4 py37h06a4308_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main widgetsnbextension 3.5.1 pypi_0 pypi wrapt 1.12.1 pypi_0 pypi xmltodict 0.12.0 pypi_0 pypi xz 5.4.2 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main zipp 3.1.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0 https://mirrors.ustc.edu.cn/anaconda/pkgs/main
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Am suffering from CUDA Error during I was running Trackformer on VisDrone2019 dataset on a server with 3 RTX3090 GPUs:
Start training /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [34,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [5,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [1,0,0], thread: [27,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed. Traceback (most recent call last): File "src/train.py", line 357, in train(args) File "src/train.py", line 284, in train visualizers['train'], args) File "/root/autodl-tmp/trackformer/src/trackformer/engine.py", line 128, in train_one_epoch loss_dict = criterion(outputs, targets) File "/root/miniconda3/envs/tf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 406, in forward losses.update(self.get_loss(loss, outputs, targets, indices, num_boxes)) File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 380, in get_loss return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs) File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 252, in loss_labels_focal losses['class_error'] = 100 - accuracy(src_logits[idx], target_classes_o)[0] RuntimeError: CUDA error: device-side assert triggeredMy conda environment is listed as below:
The text was updated successfully, but these errors were encountered: