-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Excessive memory use in 1.0.0-rc10 vs. 1.0.0-rc8? #836
Comments
Hi @fcmeyer. We're also running into memory issues lately, and it's not entirely clear why at this point. One hypothesis we have is that the main program is acquiring too much memory, so when it spawns new processes, all that memory is being copied before it can be released. Thanks for alerting us to the fact that this issue has arisen since 1.0.0-rc8. That should be useful in narrowing down the causes. |
Thanks a lot @fcmeyer, as Chris commented this is very likely a duplicate of #833. Let's keep discussion there. We are working in several directions to ease these problems (eg. #839, nipy/nipype#2284, nipy/nipype#2289). And thanks a lot for the compliments! |
One question @fcmeyer, could you please paste here the output of:
|
Hi @oesteban, sorry about the delay - was out of town for Thanksgiving. I tried running the command you listed, but it gives no output:
However, if I just do "memory", there's a SLURM error that shows the memory usage at the time the process was killed:
I hope this helps! |
@fcmeyer we have just released 1.0.0-rc12. Even though we can still see some problems with memory, I think we have tackled a great chunk of the issue. Could you test the new release out? |
@oesteban Just tried out the gold 1.0.0 release, the memory problem is definitely resolved for me. Re-running a batch of ~380 subjects in the cluster right now, about 200 done so far with no major issues or memory shutdowns from the cluster (using same settings as I had used for 1.0.0-rc8). Congrats on 1.0.0! |
Glad we prioritized on this. We'll keep trying to reduce the memory fingerprint. Thank you all for your precious feedback :) |
hey @fcmeyer could I ask you to relate your story here -> https://neurostars.org/t/fmriprep-success-stories/1111?u=oesteban ? |
@oesteban done! |
Awesome! Thanks!
…On Dec 11, 2017 3:13 PM, "fcmeyer" ***@***.***> wrote:
@oesteban <https://github.com/oesteban> done!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#836 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAkhxvQn4-xWjJRBQSN3Jfi5w8Lcl25Qks5s_YzogaJpZM4QhAW4>
.
|
Hello,
I have been testing different versions of fMRIPREP on the same two subjects in the Vanderbilt HPC using Singularity. I told SLURM to allocate 24GB to the job.
Here's the SBATCH script we used for one subject:
The version for RC8 was identical, except we changed the image to RC8 and updated the output names to have RC8 instead of RC10 for later comparison. We did this for two subjects total.
While RC8 ran with no hiccups, RC10 had two issues. First, the repetitive warning mentioned in my Neurostars post that kept spamming throughout the log:
But more concerning, SLURM killed the pipeline for both of our subjects due to excessive memory usage (I am including only the tails because the log files are super long, given the insane amount of warnings we got):
I am not sure whether the minimum memory requirements have increased since rc8, or whether this is a bug causing excessive memory usage / ignoring the limits specified in the call. But, I figured I'd bring it up in case others are experiencing this problem.
Thank you so much for developing this, I really like this tool!
The text was updated successfully, but these errors were encountered: