Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I'm a Beginner, there is some problem for me, someone can help me? thx ^_^ #50

Open
grrrrr112 opened this issue May 5, 2019 · 4 comments

Comments

@grrrrr112
Copy link

when I run the python demo.py , OOM occurred like below:
RuntimeError: CUDA out of memory. Tried to allocate 44.00 MiB (GPU 0; 1.95 GiB total capacity; 1.23 GiB already allocated; 2.62 MiB free; 98.86 MiB cached)

so where is bitch_size?

thx ^_^

@lg920810
Copy link

lg920810 commented May 6, 2019

In CornerNet.json,you should change both batch_size and chunk_sizes.For example, the batch_size is 30, the chunk_sizes is [15, 15]

@grrrrr112
Copy link
Author

In CornerNet.json,you should change both batch_size and chunk_sizes.For example, the batch_size is 30, the chunk_sizes is [15, 15]

thx for your answer.^_^

my CornerNet.json is like below:

But it's still OOM T_T

"system": {
"dataset": "COCO",
"batch_size": 1,
"sampling_function": "cornernet",

    "train_split": "trainval",
    "val_split": "minival",

    "learning_rate": 0.00025,
    "decay_rate": 10,

    "val_iter": 100,

    "opt_algo": "adam",
    "prefetch_size": 5,

    "max_iter": 500000,
    "stepsize": 450000,
    "snapshot": 5000,

    "chunk_sizes": [15, 15],

    "data_dir": "./data"
}

[root@localhost CornerNet-Lite]# python3 demo.py
total parameters: 116969339
loading from /data/MachineLearning/CornerNet-Lite/core/../cache/nnet/CornerNet_Saccade/CornerNet_Saccade_500000.pkl
/usr/local/python3/lib/python3.7/site-packages/torch/nn/functional.py:2423: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode))
Traceback (most recent call last):
File "demo.py", line 10, in
bboxes = detector(image)
File "/data/MachineLearning/CornerNet-Lite/core/base.py", line 25, in call
bboxes = self._inference(image, *args, **kwargs)
File "/data/MachineLearning/CornerNet-Lite/core/base.py", line 21, in _inference
return self._func(self._db, self._nnet, image.copy(), *args, **kwargs)
File "/data/MachineLearning/CornerNet-Lite/core/test/cornernet_saccade.py", line 346, in cornernet_saccade_inference
dets, atts = decode_func(db, nnet, images, no_att=no_att)
File "/data/MachineLearning/CornerNet-Lite/core/test/cornernet_saccade.py", line 77, in batch_decode
test=True, num_dets=num_dets, no_border=True, no_att=no_att
File "/data/MachineLearning/CornerNet-Lite/core/nnet/py_factory.py", line 112, in test
return self.model(*xs, **kwargs)
File "/usr/local/python3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/data/MachineLearning/CornerNet-Lite/core/nnet/py_factory.py", line 30, in forward
return self.module(*xs, **kwargs)
File "/usr/local/python3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/data/MachineLearning/CornerNet-Lite/core/models/py_utils/modules.py", line 303, in forward
return self._test(*xs, **kwargs)
File "/data/MachineLearning/CornerNet-Lite/core/models/py_utils/modules.py", line 280, in _test
cnvs, ups = self.hg(image)
File "/usr/local/python3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/data/MachineLearning/CornerNet-Lite/core/models/py_utils/modules.py", line 234, in forward
inter = self.intersind
File "/usr/local/python3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/data/MachineLearning/CornerNet-Lite/core/models/py_utils/utils.py", line 194, in forward
return self.relu(bn2 + skip)
RuntimeError: CUDA out of memory. Tried to allocate 44.00 MiB (GPU 0; 1.95 GiB total capacity; 1.21 GiB already allocated; 20.62 MiB free; 75.49 MiB cached)
[root@localhost CornerNet-Lite]#

@lg920810
Copy link

lg920810 commented May 7, 2019

How many GPU do you have? If you have just one, I think you can try set chunk_sizes=[1].

@grrrrr112
Copy link
Author

grrrrr112 commented May 7, 2019

How many GPU do you have? If you have just one, I think you can try set chunk_sizes=[1].

OOM T_T

But if I use debug mode to execute setp by setp, it can be done.

Maybe I should change my video card >_<

[root@localhost ~]# nvidia-smi
Tue May 7 20:31:20 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 750 Ti Off | 00000000:01:00.0 On | N/A |
| 38% 28C P8 1W / 38W | 222MiB / 1999MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 12655 G /usr/bin/X 109MiB |
| 0 13192 G /usr/bin/gnome-shell 110MiB |
+-----------------------------------------------------------------------------+
[root@localhost ~]#
Screenshot from 2019-05-07 20-34-18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants