2D Classification Stalling on GPU #166
Comments
|
Original comment by Dari Kimanius (Bitbucket: dkimanius, GitHub: dkimanius): This is a rare issue that I haven't observed mush since the last fix. It seem to occur for odd images when the prior and likelihood functions do not overlap within the single precision limits. Till we fix this issue please do try a different setting for the run. One flag that you may want to try is the maxsig-flag. In 2D you can set the maximum number of significant coarse weights to for instance 500 by providing the additional flag "--maxsig 500". |
|
Original comment by Dari Kimanius (Bitbucket: dkimanius, GitHub: dkimanius): Hi @barureddy, did this get resolved? If not please try a recent patch for this particular issue (attached). To apply it, cd into your local repository and run: Remember to update your repository with the latest verison first. |
|
Original comment by Dari Kimanius (Bitbucket: dkimanius, GitHub: dkimanius): You'll have to download the attached file and refer to its path. If you download it to the Download directory for instance write: Where "HOME" is the path to you home directory. |
|
Original comment by Dari Kimanius (Bitbucket: dkimanius, GitHub: dkimanius): It ran for at least 10 iterations, so you should start seeing at least a rough shape of your particle. Have a look at the classes of the last iteration and check whether you see anything. If there's only noise, the refinement is not converging for some reason. |
|
Original comment by Robert McLeod (Bitbucket: robbmcleod, GitHub: robbmcleod): Hi, We have similar issues on some quad Titan-XP nodes. What happens is that two GPUs finish, and then 1-2 seem to stall. The time estimate for Relion to finish the iteration reaches the estimate and then continues forever. I am struggling, however, on how to get some sort of useful debugging info out of the situation. This job was launched with Relion 2.02 patched with the double-mode fix you supplied above. All CPU threads continue to run at 100 %. We had similar troubles with 2.01. The particles are somewhat elongated by this iteration but still featureless. I could probably make a tarball and provide a download link. The particles look to be 31 GB. I'll also ask the graduate student to re-extract with tighter particle picking conditions and see if that helps. However the particle picking looks good to my eye. |
|
Original comment by Dari Kimanius (Bitbucket: dkimanius, GitHub: dkimanius): Hi @robbmcleod, Considering that you're not getting any error messages, I doubt this is the same issue addressed by the provided fix. What happens if you run with only 2 GPUs, do you see a similar behaviour? Do you have other GPU-systems you can reproduce this on? If the issue doesn't occur on our systems there's unfortunately no way it can be directly addressed. |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): I have experienced a similar problem to Robert on my system with 3 Titan X Maxwells when I run two of them. Relion stalls at the end of iteration 2 without any error and one of the GPUs holds data in memory but doesn't appear to be doing anything and the run.out doesn't write anything after shortly before the expected end of the iteration. I paste some monitoring data (nvidia-smi, sensors and mpstat) followed by the run.out, which also shows that I can continue the run and complete and start iteration 3. I have not seen this issue using when using all three GPUs but I have also seen the following error in run.out files in both cases (which I paste before the monitoring for the stalling). It should probably be noted that the all three GPUs case has allowed me to run a full set of 25 iterations. All these runs are using the Benchmark dataset. Expectation iteration 2 of 25
0.55/28.93 min . 2.20/34.05 min ... 3.88/35.23 min ...... 5.55/35.62 min ......... 7.22/35.82 min ............ 8.90/36.02 min .............. 10.58/36.15 min ................. 12.27/36.25 min .................... 13.93/36.28 min ....................... 15.62/36.35 min ......................... 17.28/36.37 min ............................ 18.93/36.35 min ............................... 20.58/36.33 min ................................. 22.25/36.35 min .................................... 23.92/36.37 min ....................................... |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): I forgot to mention that I eventually had to kill the job when it had not done something for 20-30 minutes and then I continued it from the end of iteration 1. |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): For 3 GPUs, I have -np 4 and --j 4 --gpu "0:1:2" In both cases, I am specifying which GPU for all slave ranks and the assignment is showing up correctly in the run.out files. |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): Also to help interpret that, I should mention that my system maps GPUs wrong so if I ask for the GPU with ID 0 then nvidia-smi shows that GPU #2 has been activated. The same happens when running gromacs-2016. |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): I just thought that I'm not using the latest beta version necessarily - I am using relion-devel-lmb with the cmake-3.7 patch applied. I am now installing the latest relion2-beta, which I cloned from git a few days ago and applied the cmake-3.7 patch to as well. It could be that I am reporting an issue that has already be resolved and in that case sorry. |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): ok I cloned and installed relion2-beta again to be sure. It's not running fine and has reached iteration 3 with 2 GPUs so it looks like I did hit a fixed issue. |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): Actually this issue is still there. It now got the following error at the start of iteration 14: Expectation iteration 13 of 25
Maximization ... 21/ 21 sec ............................................................ |
|
Original comment by Dari Kimanius (Bitbucket: dkimanius, GitHub: dkimanius): Hi @james_krieger! Please read the original issue description next time before hijacking it. Your issue is not related to this one and will certainly make things confusing for future references. Post this report in a new issue and attach the (small sized) output files from the run. |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): Hi Dari, I did read the original issue and I thought initially that it was the same. I will post a new issue and attach the files there as you say. Best wishes |
|
Original comment by james_krieger (Bitbucket: james_krieger, GitHub: Unknown): Hi Dari, I have now created a new issue (#189), which I hope is ok. Best wishes |
|
Original comment by Özkan Yildiz (Bitbucket: oeyildiz, GitHub: Unknown): Our current status to this problem is the following: The problem appears to be more likely with more pooled particles. @robert McLeod, how many particles did you pool for the run you posted here? |
|
Original comment by Robert McLeod (Bitbucket: robbmcleod, GitHub: robbmcleod): Typically 100. These nodes have 512 GB of system memory so a lack of allocatable memory should not be an issue in our case. |
|
Original comment by Özkan Yildiz (Bitbucket: oeyildiz, GitHub: Unknown): Could you repeat the same run as above with --pool 1 and check its behaviour? |
|
The original issue (stalling execution during last phase of an expectation step) is fixed in v2.0.4. It resulted from using threads in an inherently unsafe manner within the cuda-context. During fast enough runs, this led to occasional deadlock in cuda-internal pthread-mutex locks. This also explins why decreasing pool helps; it decreases thread-based parallelism and the concurrency which caused the stall. |
|
I have been hitting something which could be this issue, while running relion (2.0.3/openmpi 1.10.6/CentOS-6) benchmaks on a single 16 cores/2 sockets 4x titanX pascal: random (non reproduceable) stalled jobs 100% cpus/gpus (1/2/4)... |
|
Sounds like it is. Updating the code according to the instructions on the landing page should eliminate this. |
|
Hi |
|
|
|
Hello @bforsbe , I am using Relion 2.1.0 with OpenMPI 3.0.0 and I believe I am having all the symptoms of this problem. When I run large 2D classifications (800,000 particles, box size 320), I get hang ups on one GPU as described above while the others are inactive. This problem seems to be stochastic, as I have run a very similar job with the same particles, and it went through 25 iterations. However, after it happens once and I cancel the job, subsequent jobs seem to recapitulate the problem, and for a few days, many jobs I run with more than 50k or 100k particles get stuck at one of the expectation iterations. There is no error message, and the job hangs up indefinitely. I am currently trying the a job that just suffered from this problem with the --pool 1 option to see if it helps. I know you said this problem was fixed as of v2.0.4, but I am still struggling with it. Any advice? Thanks! |
|
Did you try rebooting the computer? Sounds like a memory (RAM) issue. |
|
@bforsbe Thanks for the quick reply. I will have system administrator reset our GPU nodes tomorrow to see if it helps, and I'll get back to you. To help my understanding, could you perhaps expand on how a RAM issue would cause this type of problem? |
|
RELION is not completely deterministic in its results, but getting any error completely stochastically rather hints at some hardware-flaw that you hit. It could still be in the algorithm, but a lot of people are using the algorithm, and only you are using your hardware. That being said, I don't actually know what error you are getting. This is a long and convoluted thread and "as the above" is unfortunately not very descriptive, in part because at least one false observation was given at some point. |
|
@bforsbe My apologies for the lack of clarity. I believe my issue is more similar to #53 and #54 (please let me know how to move my comments if that would help clarity). It appears in issue #53 that Ozkan Yildiz had the same problem I was having; namely, the 2D classification would stall at the very last part of the expectation step with no error message. 1 GPU would show 100% utilization (although low wattage) while the others have 0% utilization and no memory used. Ozkan seemed to have solved this problem by using a different data set before the discussion diverged into talking about GPU indexing... Although this problem seemed to manifest again for Ozkan (comment at the end of issue #53), and was solved by omitting the flag --dont_combine_weights_via_disc in Relion 1.4. However, Ozkan comments on December 16, 2016 (issue #54) that the problem has resurfaced and that they resorted to using CPU's to avoid this problem all together (comment in issue #54 towards middle of page). Finally, Ozkan mentioned that using --pool 1 alleviated the stalling GPU problem. In issue #54, @cdk reported that they were having a problem with a job crashing, yet a GPU still running. This would lead users to starting new jobs and getting memory allocation errors. However, @cdk did not follow-up with much information regarding the current state of that problem for them. At the end of #54, @bforsbe stated that "Using v2.0.4 you should no longer find zombies, as the issue was identified and fixed. It was caused by not-so cautious use of htread-parallelism in a single cuda-context, which in turn caused some dead-locks in internal cuda-threads. The zombie was thus a spin-lock in cuda waiting to return an API-call." However, I am not sure if there is a difference between a zombie still running on the GPU after a job "crashes" and what I am seeing where the job never technically "crashes," but remains stuck at the last part of an expectation step indefinitely with one GPU at 100% and all others going to 0%. I have re-started our GPU nodes, and re-run the job starting at the iteration immediately before the crash (iteration 3). The job is now on iteration 6 and seems to be operating successfully. I tracked the buff/cache memory over the course of the run using the command free as I was curious about the RAM. After reboot, buff/cache was empty, and over the several iterations it is now close to ~114-120 GB on all the GPU nodes (we have 128 GB of RAM on each GPU node). Thank you for your tip regarding rebooting the machines. Is there any other information I can provide to you so that we may nail down the problem? I have run many successful 2D and 3D classifications; however, this problem is the only one I get in Relion that doesn't produce an error message and leaves me sort of clueless as to what is happening. Any help is appreciated. Thanks! |
|
I'm pretty certain your issue is not a lockup as fixed in issue #54, just to have that said. Relion dispatches mpi-slaves and waits for them to return before continuing. Anytime a slave get stuck doing anything, the symptoms are largely the same, which is why many issues look the same. The interesting observation in your case is the 100% gpu util. Now, that does not mean that they are working hard, just that the slave got stuck in a state where the gpu was busy. So network dropouts or RAM-swapping is actually quite unlikely. Thinking about it again, the gpu-driver or cuda-runtime are more likely to have dropped out or ended up in a weird state. The first debug for this is, yep, a reboot. I don't like to say that this is the issue, because it's not something I can fix. But it's looking likely, unless you have further issues. Let me know if that's the case and we'll try to dig deeper. Thanks for the simply splendid clarification! Your in my good book. |
|
@bforsbe Thanks for the reply! No problem, writing the summary of the other issues actually helped me clarify the issue myself. Just for completeness, I will post my newest set of experiments that may help confirm the issue. All of these experiments were run across 7 nodes, each with one GPU card (2 1080's and 5 1080 Ti's), using mpi and 8 mpi processes per card (except the first node which has 7 procs per GPU due to mpi master being on this node; I am using SLURM, therefore the first cpu gets assigned to master, and the other seven go to slaves on node 1). The particles.star file contained ~800k particles with a box size of 320 pixels (1.07 A/pix). To determine if rebooting computers solved the problem, we rebooted all the nodes and ran my 2D classification. The result was that the job ran successfully through 25 iterations. To determine if having buffer/cache "full" at start of job is causing problems versus some cuda or gpu-driver problem, we re-ran the same 2D classification without doing anything additional to the computers. The buff/cache memory was almost full (~115GB on 128 GB RAM nodes) at the start of the job, and I checked to make sure there were no zombie processes on any node before running the job. The result was that the classification hangs at the end of iteration 4 with all the usual symptoms (2 of the 1080 Ti's stalling at end of expectation with 100% GPU util, 75W/280W, and 10Gb/11Gb GPU memory used. All other GPU's are empty, 0% util, no memory used, etc.). To further investigate the role of RAM in this problem, we cleared the caches without rebooting the nodes and performed the same classification (again, checking for any sort of zombie processes). The result was that the classification hangs at the end of expectation 2. This time with a different 1080 Ti hanging at 100% util. (Note: I have performed similar experiments, and it is never the same GPU that stalls at the end of the expectation, the problem appears to happen indiscriminately among 1080's and 1080 Ti's among the nodes). Does this to you vindicate any sort of RAM issue causing the problem and place the blame again on the gpu-driver or cuda-runtime doing something weird at the last part of expectation step? Any additional thoughts? Thanks! |
|
8 MPI-ranks per GPU is more than I have ever run. For 2D-classification, you CAN of course do that because the memory footprint of 2D-images is so much less than that of 3D-models, but I would think that there's not much performance to get from having that many. I can try to reproduce by simply crowding a GPU with an insane number of MPIs and see if there's some contention or race-condition we missed across MPIs. My advice in the meantime is to scale back to one or two MPIs per GPU and pack on the threads instead. For 2D, I tend to just
With those settings, most things fly through pretty darn quick, even on one GPU. It avoids MPI-communication overhead and RAM-caches everything. Gets the job done. Some choose to linger and perfect their 2D-classes, but this is time wasted in my mind, especially for initial cleaning of otherwise good data. |
|
@bforsbe Thanks for the suggestions! I talked to a very experienced colleague yesterday, and he says he usually has does 3-4 MPI procs per GPU for 2D with no problems (but he definitely did not like the idea of 8 mpi per gpu!). I will definitely lower the number of mpi procs per GPU in the future, and try your other suggestions. |
|
Hi @bforsbe , I am wondering if there is a way to include "-G" (big G) option for nvcc for cuda kernel debugging? I am still debugging this issue. It doesn't seem that DCMAKE_BUILD_TYPE=Debug enables -G for nvcc unless I'm mistaken... My current workaround is to use --continue when jobs get stuck on an iteration. This usually pushes them through until they finish, or if they get stuck again at a later iteration, then I do --continue again until all iterations are done. |
|
read line 5 of Cmake/BuildTypes.cmake. Of note, you might also have to set |
|
Hi guys, I hope you doing well all, I new on using RELION 3.1, I'm facing some issues on using 2D classification on RELION 3.1 I did follow the installation and try to run the GUI it did work but when I tried to run the 2D classification it did not work, I wen back to the comment line approach and download the benchmark dataset with the STAR file and when I tried to run the following comment: mpirun -n XXX I got the following error: please could someone help me how to run the 2D classification in RELION 3.1 really appreciate that! |
|
Hi, you cannot use the XXX when you run from the command line. You need to
specify an actual number of mpi tasks. XXX type variables only work when
you are using a template script for the Relion GUI to fill in the variables
with values from the gui.
…On Sat, Sep 26, 2020 at 1:40 PM adil-20 ***@***.***> wrote:
Hi guys, I hope you doing well all, I new on using RELION 3.1, I'm facing
some issues on using 2D classification on RELION 3.1 I did follow the
installation and try to run the GUI it did work but when I tried to run the
2D classification it did not work, I wen back to the comment line approach
and download the benchmark dataset with the STAR file and when I tried to
run the following comment:
mpirun -n XXX which relion_refine_mpi --i Particles/shiny_2sets.star
--ctf --iter 25 --tau2_fudge 2 --particle_diameter 360 --K 200 --zero_mask
--oversampling 1 --psi_step 6 --offset_range 5 --offset_step 2 --norm
--scale --random_seed 0 --o class2d
I got the following error:
[image: Screenshot from 2020-09-26 15-38-19]
<https://user-images.githubusercontent.com/71946383/94349869-69d45f80-000e-11eb-9aca-8587195397a3.png>
please could someone help me how to run the 2D classification in RELION
3.1 really appreciate that!
@bforsbe <https://github.com/bforsbe>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#166 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGDIR24FULQUNE4VOEFCC7DSHZGTPANCNFSM4DCTAVOA>
.
|
Originally reported by: Bharat Reddy (Bitbucket: barureddy, GitHub: barureddy)
When I run 2D Classification using my gpus I get an error with relion stalling before it reaches the 25 iterations. The master thread still runs in a zobie like process. My issue looks like #24, but I am not sure. The particle files are quite large, but I can upload them to you if needed.
The text was updated successfully, but these errors were encountered: