-
Notifications
You must be signed in to change notification settings - Fork 222
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CUDA timeout in preprocessing, KS3 #323
Comments
Can you please provide the command line output? Something seems to be going wrong. In the past we've had to disable that watchdog, but I think current Nvidia drivers automatically disable it or circumvent it somehow. Maybe it's different for Quadro cards. There shouldn't be a disadvantage to disabling the watchdog, and your screen response should not become slow. |
First it give me this repeating warning that recurs for nearly how much space there is in the command window
then it throws this error
My CUDA version is 10.2. My colleague who can run KS 3 without any issues has a Quadro P4000. Maybe mine is just a bit underpowered to handle the task? |
8GB of gpu RAM is more than enough. I forgot to mention, but you also need to install the specific version of CUDA that your Matlab version requires: https://www.mathworks.com/help/parallel-computing/gpu-support-by-release.html;jsessionid=f68ff768914bd294d61356fc7d1d |
Yeah, like I mentioned before, colleagues of mine get by easily with a P4000, which has nearly he same specs. I noticed that there were 5 (!) versions of CUDA installed on the PC so removed all of them and left CUDA 10.0. I am working with Matlab version 2019b so it should be the right one. I am attaching a screenshot from the task manager where KS 3 is preprocessing NP data. You can see how it ramps up to 100% on every chunk that is processed. |
The memory usage is stable, that's just the usage ramping up. Your GPU really is up to the task, but there must be something wrong with it's configuration. Have you updated the Nvidia drivers? This is separate from CUDA. In cases like this I would just start over from scratch with uninstalling and re installing visual studio, CUDA and Matlab, in that order. |
I reinstalled CUDA and VisualStudio but still have the same issue. Though the throttling pattern of the CPU changed, which shows something changed under the hood (screenshot attached).
|
Hi,
when I am preprocessing Neuropixel phase 3a or NP 1.0 data in KiloSort 3 I get an error from CUDA in the KS GUI. This error happens after the screen response has lagged for a few minutes and I haven't been able to do any work in parallel.
CUDA_ERROR_LAUNCH_TIMEOUT
I have looked up the error online and it seems to be due to the fact that my graphics card (Nvidia Quadro M4000) has to serve my screen and KS at the same time. When the KS instruction to the GPU takes longer than 2 seconds to complete a protocol is triggered that resets the graphics driver (and cancelling KS). It would be possible to remove 2 second threshold in the Windows regEdit but this would only alleviate the symptom and not solve the problem (while additionally making the screen response slow).
What I have tried so far:
I have seen that when KS 3 is preprocessing data the graphics card is occupied only period-wise. Though KS 3 has a new spike detection algorithm it looks like it is processing chunks like KS 2. A colleague of mine can run KS 3 fine with a Quadro P4000 without any frame rate drops, which only slightly better than my M4000.
The text was updated successfully, but these errors were encountered: