-
-
Notifications
You must be signed in to change notification settings - Fork 928
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Check failed: error == cudaSuccess (2 vs. 0) out of memory #28
Comments
For your reference, just in case.
using Python 3.7.1
Thanks |
Thank you for the issue. This looks weird. I saw the issue this morning but has yet located the problem. Just know that I've already see the problem and is working on it. There's a big update coming soon. This issue might have to be fixed after the update which is already in process. |
Thanks for your quick reply! |
Just hit this right now. Hopefully you'll be able to figure it out |
This also happens to me, even after reducing the amount of threads from 5 to 4, because I have insufficient memory available |
Could you please monitor the system to make sure that there's sufficient memory available for both the system RAM and the GPU RAM? Btw I stuffed your output in a code block to make it easier to read. |
I was just reading some other similar errors, but I'm still not sure how to fix this problem on my side.
Some other caffe-library-related issues as a reference: |
I followed your suggestions and it turns out that my system is running out of memory. I had to limit thread count to 2 threads on my particular test to get it to run. My laptop has a GTX 965M and it only has 4GB of VRAM. System memory was more than enough with over 11 GB of RAM free. |
@asimonf so that is indeed the problem. My program is able to monitor the system memory and give suggestions on how many threads should be used, and warns the user when insufficient memory is available. However, this program doesn't have the capability to monitor GPU memory, which reflects as a CUDA error in @chiwing4 's ouput as "cudaSuccess out of memory". I remember that I've already looked into that, but didn't find an elegant solution for monitoring cuda GPU memory. I will keep looking into it. Therefore, for now, if this issue arises, reduce the number of threads. |
My current testing with your file shows around 1.4 GB of memory per thread. I don't know if memory usage grows with output size, but if it does, it might even more (testing output at 1440x1080). I'll test a bit more. A good suggestion could be to limit threads assuming a maximum usage of 2GB per thread? |
Maybe it's possible. I'll have to look into if waifu2x-caffe has any options for that. |
OK I can confirm this is GPU out of memory problem. GPU memory usage
My testing show ~4GB VRAM per thread. Output to 1080p and 4K has the same memory usage. |
I just enlarged the testvid.mp4 (240 frames, 320 x 240) to 1080p and 4K.
4K needs nearly 4x computational time than 1080p. (Interesting. 4K has 4 times more pixels than 1080p) |
Interesting. I thought it would be much faster. I tried an OpenCL version on my house computer (It has a Vega 56 card) and it took about that long for the 1080 one. I might not be remembering accurately, so I'll try testing again tonight. |
@asimonf I thought waifu2x only supports cuda? Maybe I'm wrong on that. I don't have an AMD card to test things out. |
Yes, but there's a rewrite in C++ that is compatible with CUDA and OpenCL. I haven't tested enough to see if quality is similar. I just tested speed yesterday. It might not be the same (but it does use the same models for the NN though, so it can't be that different). Here's the link: https://github.com/DeadSix27/waifu2x-converter-cpp. I doubt it's faster than the original waifu2x on nvidia cards, but I don't know. |
That's cool. It's fascinating how many derivations there are from the original waifu2x. I feel like this thread is beginning to be like a forum. |
So I just tested it on the Vega 56 and it took 432 seconds to process all 240 frames with upscale and denoise to 1440x1080. I can't really compare it 100% to what waifu2x-caffe does with video2x, but apparently it isn't that slow compared to the 2080ti if (and this is a big if) they are working to generate similar quality images. |
So with 2 threads, this works for me too. But I have 2 1080Ti. So can I spread the load on both the GPUs? |
@dealvidit spreading the load will unlikely be a thing that this program can control. It's more up to either waifu2x or caffe2. |
I'm getting some progress. I'm considering using the |
I have just pushed update Although it doesn't have support for AMD GPUs, AMD GPUs must use |
K4YT3X Edit: Temporary Solution
The issue is caused by waifu2x-caffe not having sufficient memory. A temporary solution to this problem is to reduce the number of threads used.
Original Issue
Hi, the program failed and I am having the following exception when I use --gpu to enlarge.
I believe my machine has enough memory to run it.
It there any way to fix it?
Many thanks. This is a useful program.
The text was updated successfully, but these errors were encountered: