Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError: CUDA out of memory #1

Closed
wjs20 opened this issue Oct 21, 2022 · 1 comment
Closed

RuntimeError: CUDA out of memory #1

wjs20 opened this issue Oct 21, 2022 · 1 comment

Comments

@wjs20
Copy link

wjs20 commented Oct 21, 2022

Hi

Thanks for uploading the code, this looks really interesting!

I have had some memory issues running your example, maybe you can help me?

it comes when I run the model.generate(epitope, surface) function

out = model.generate(epitope, surface)

RuntimeError: CUDA out of memory. Tried to allocate 120.00 MiB (GPU 0; 4.00 GiB total capacity; 3.09 GiB already allocated; 0 bytes free; 3.34 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I've put my system details below

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.81       Driver Version: 513.29       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA T500         On   | 00000000:01:00.0 Off |                  N/A |
| N/A   43C    P8    N/A /  N/A |   3848MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2175      C   /python3.9                      N/A      |
+-----------------------------------------------------------------------------+

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
alsa-lib                  1.2.7.2              h166bdaf_0    conda-forge
anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py39hb9d737c_2    conda-forge
asttokens                 2.0.8              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
biopython                 1.79             py39hb9d737c_2    conda-forge
blas                      2.116                       mkl    conda-forge
blas-devel                3.9.0            16_linux64_mkl    conda-forge
bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py39hb9d737c_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py39he91dace_1    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
contourpy                 1.0.5            py39hf939315_0    conda-forge
cryptography              38.0.2           py39hd97740a_0    conda-forge
cudatoolkit               11.7.0              hd8887f6_10    nvidia
cudnn                     8.4.1.50             hed8a83a_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.6.3            py39h5a03fae_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 1.1.1              pyhd8ed1ab_0    conda-forge
expat                     2.4.9                h27087fc_0    conda-forge
fftw                      3.3.10          nompi_hf0379b8_105    conda-forge
flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               hc2a2eb6_1    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.37.4           py39hb9d737c_0    conda-forge
freetype                  2.12.1               hca18f0e_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
glib                      2.74.0               h6239696_0    conda-forge
glib-tools                2.74.0               h6239696_0    conda-forge
gst-plugins-base          1.20.3               h57caac4_2    conda-forge
gstreamer                 1.20.3               hd4edc92_2    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4           py39hf3d152e_0    conda-forge
importlib_resources       5.10.0             pyhd8ed1ab_0    conda-forge
ipykernel                 6.16.1             pyh210e3f2_0    conda-forge
ipython                   8.5.0              pyh41d4057_1    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.0.2              pyhd8ed1ab_1    conda-forge
jack                      1.9.21               h2a1e645_0    conda-forge
jedi                      0.18.1             pyhd8ed1ab_2    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
jq                        1.6               h36c2ea0_1000    conda-forge
jsonschema                4.16.0             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0            py39hf3d152e_7    conda-forge
jupyter_client            7.4.3              pyhd8ed1ab_0    conda-forge
jupyter_console           6.4.4              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py39hf3d152e_0    conda-forge
jupyter_server            1.21.0             pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.3              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py39hf939315_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.39                 hc81fddc_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0            16_linux64_mkl    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcap                    2.66                 ha37c62d_0    conda-forge
libcblas                  3.9.0            16_linux64_mkl    conda-forge
libclang                  14.0.6          default_h2e3cab8_0    conda-forge
libclang13                14.0.6          default_h3a83d3e_0    conda-forge
libcups                   2.3.3                h3e49a29_2    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.1                h27087fc_0    conda-forge
libgcc-ng                 12.2.0              h65d4601_18    conda-forge
libgfortran-ng            12.2.0              h69a702a_18    conda-forge
libgfortran5              12.2.0              h337968e_18    conda-forge
libglib                   2.74.0               h7a41b64_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0            16_linux64_mkl    conda-forge
liblapacke                3.9.0            16_linux64_mkl    conda-forge
libllvm14                 14.0.6               he0ac6c6_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.38               h753d276_0    conda-forge
libpq                     14.5                 hd77ab85_0    conda-forge
libprotobuf               3.21.8               h6239696_0    conda-forge
libsndfile                1.1.0                h27087fc_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.39.4               h753d276_0    conda-forge
libstdcxx-ng              12.2.0              h46fd767_18    conda-forge
libtiff                   4.4.0                h55922b4_4    conda-forge
libtool                   2.4.6             h9c3ff4c_1008    conda-forge
libudev1                  249                  h166bdaf_4    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libuv                     1.44.2               h166bdaf_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvm-openmp               14.0.4               he0ac6c6_0    conda-forge
magma                     2.5.4                hc72dce7_3    conda-forge
markupsafe                2.1.1            py39hb9d737c_1    conda-forge
matplotlib                3.6.1            py39hf3d152e_0    conda-forge
matplotlib-base           3.6.1            py39hf9fd14e_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   2.0.4              pyhd8ed1ab_0    conda-forge
mkl                       2022.1.0           h84fe81f_915    conda-forge
mkl-devel                 2022.1.0           ha770c72_916    conda-forge
mkl-include               2022.1.0           h84fe81f_915    conda-forge
mpg123                    1.30.2               h27087fc_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.31               haf5c9bc_0    conda-forge
mysql-libs                8.0.31               h28c427c_0    conda-forge
nbclassic                 0.4.5              pyhd8ed1ab_0    conda-forge
nbclient                  0.7.0              pyhd8ed1ab_0    conda-forge
nbconvert                 7.2.2              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.2.2              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.2.2              pyhd8ed1ab_0    conda-forge
nbformat                  5.7.0              pyhd8ed1ab_0    conda-forge
nccl                      2.14.3.1             h0800d71_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
notebook                  6.5.1              pyha770c72_0    conda-forge
notebook-shim             0.2.0              pyhd8ed1ab_0    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.78                 h2350873_0    conda-forge
numpy                     1.23.4           py39h3d75532_0    conda-forge
oniguruma                 6.9.8                h166bdaf_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1q               h166bdaf_1    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandoc                    2.19.2               ha770c72_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.37                hc3806b6_1    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0            py39hd5dbb17_2    conda-forge
pip                       22.3               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
prometheus_client         0.15.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.31             pyha770c72_0    conda-forge
prompt_toolkit            3.0.31               hd8ed1ab_0    conda-forge
psutil                    5.9.3            py39hb9d737c_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio                14.0                habe0971_10    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7           py39h18e9c17_1    conda-forge
pyqt5-sip                 12.11.0          py39h5a03fae_1    conda-forge
pyrsistent                0.18.1           py39hb9d737c_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.9.13          h9a8a25e_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.12.1          cuda112py39hb0b7ed5_201    conda-forge
pyzmq                     24.0.1           py39headdf64_0    conda-forge
qt-main                   5.15.6               hc525480_0    conda-forge
qtconsole                 5.3.2              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.3.2              pyha770c72_0    conda-forge
qtpy                      2.2.1              pyhd8ed1ab_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                65.5.0             pyhd8ed1ab_0    conda-forge
sip                       6.7.2            py39h5a03fae_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.39.4               h4ff8645_0    conda-forge
stack_data                0.5.1              pyhd8ed1ab_0    conda-forge
tbb                       2021.6.0             h924138e_0    conda-forge
terminado                 0.16.0             pyh41d4057_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
torchvision               0.13.0          cuda112py39hd2c45b6_0    conda-forge
tornado                   6.2              py39hb9d737c_0    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.5.0              pyhd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022e                h191b570_0    conda-forge
unicodedata2              14.0.0           py39hb9d737c_1    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.4.1              pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.3              pyhd8ed1ab_0    conda-forge
xcb-util                  0.4.0                h166bdaf_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h166bdaf_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h166bdaf_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.9.0              pyhd8ed1ab_0    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge
@wjs20
Copy link
Author

wjs20 commented Oct 22, 2022

It seemed to be an issue with my pytorch installation, I've tweaked it a bit and seems to be working now.

Closing issue.

@wjs20 wjs20 closed this as completed Oct 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@wjs20