Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It seems like out of memory #34

Closed
lfxxxxx opened this issue Apr 2, 2017 · 3 comments
Closed

It seems like out of memory #34

lfxxxxx opened this issue Apr 2, 2017 · 3 comments

Comments

@lfxxxxx
Copy link

lfxxxxx commented Apr 2, 2017

[lfx@gpu-com caffe_rtpose-master]$ ./build/examples/rtpose/rtpose.bin --image_dir ~/caffe_rtpose-master/image/
./build/examples/rtpose/rtpose.bin: /home/lfx/tools/anaconda2/lib/libtiff.so.5: no version information available (required by /usr/local/lib/libopencv_imgcodecs.so.3.1)
F0402 10:24:15.074151 25048 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f7d4a812e6d (unknown)
@ 0x7f7d4a814ced (unknown)
@ 0x7f7d4a812a5c (unknown)
@ 0x7f7d4a81563e (unknown)
@ 0x7f7d519eaed2 caffe::SyncedMemory::to_gpu()
@ 0x7f7d519ea249 caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f7d51863e62 caffe::Blob<>::mutable_gpu_data()
@ 0x7f7d51a60e7f caffe::CuDNNConvolutionLayer<>::Forward_gpu()
@ 0x7f7d519b2f2b caffe::Net<>::ForwardFromTo()
@ 0x40bdff warmup()
@ 0x411e3b processFrame()
@ 0x7f7d3eb95dc5 start_thread
@ 0x7f7d3e8c473d __clone
Aborted (core dumped)

[lfx@gpu-com caffe_rtpose-master]$ nvidia-smi
Sun Apr 2 10:25:18 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26 Driver Version: 375.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GT 730 Off | 0000:01:00.0 N/A | N/A |
| 40% 59C P8 N/A / N/A | 2MiB / 978MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX TIT... Off | 0000:06:00.0 Off | N/A |
| 53% 83C P2 184W / 250W | 4192MiB / 12206MiB | 97% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
| 1 12602 C python 139MiB |
| 1 22669 C python 139MiB |
| 1 23028 C /home/xhq/caffe/build/tools/caffe 3903MiB |
+-----------------------------------------------------------------------------+

I have no idea why still have the error "out of memory" even test in just 1 image.
[lfx@gpu-com caffe_rtpose-master]$ ls image
image1.jpg

@gineshidalgo99
Copy link
Member

gineshidalgo99 commented Apr 2, 2017

Hi, you need around 2GB of free memory on your GPU to run our code. You can check that with the bash command: nvidia-smi. Are you sure you are using the GPU with 12GB instead of the one of only 1GB?
Another thought is that you compiled without cuDNN. Without cuDNN, it requires more than 12GB of GPU memory. Let us know if any of this was your problem, thanks! Otherwise, re-open this issue please.

@lfxxxxx
Copy link
Author

lfxxxxx commented Apr 7, 2017

I have compiled rtpose-caffe with cuDNN. (I have set USE_CUDNN:=1), but still have this error

[lfx@gpu-com rtpose-caffe]$ .build_release/examples/rtpose/rtpose.bin --image_dir image/
F0407 10:27:17.102958 1129 syncedmem.cpp:64] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7efeaa736e6d (unknown)
@ 0x7efeaa738ced (unknown)
@ 0x7efeaa736a5c (unknown)
@ 0x7efeaa73963e (unknown)
@ 0x7efeb1905912 caffe::SyncedMemory::to_gpu()
@ 0x7efeb1904c79 caffe::SyncedMemory::gpu_data()
@ 0x7efeb177e762 caffe::Blob<>::gpu_data()
@ 0x7efeb197b9d3 caffe::CuDNNConvolutionLayer<>::Forward_gpu()
@ 0x7efeb18cdb8b caffe::Net<>::ForwardFromTo()
@ 0x40bcff warmup()
@ 0x411cbb processFrame()
@ 0x7efe9ed99dc5 start_thread
@ 0x7efe9eac873d __clone
Aborted (core dumped)

@gineshidalgo99
Copy link
Member

I will answer in #37. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants