Issue preprocessing multiple multi echo / multi band data with fmriprep update #2728
Comments
|
This could be an issue from using a more recent version of tedana. @tsalo could I bug you to take a look? |
|
It may also be that we need |
|
It looks like tedana's memory usage is the problem here. Two options I could see are:
|
|
Is there anything I can do to help test, next steps, etc.? Thanks so much for taking a look at this issue - obviously we don't want to have to preprocess an entire dataset one participant at a time! |
|
@njr175 I've opened an issue in the tedana repo (ME-ICA/tedana#856), but it will probably be a while before we can propagate any changes to an fMRIPrep release, since we'll need to (1) fix the problem on tedana's side, (2) make a new release, (3) pin the new version in fMRIPrep's requirements, and (4) make a new fMRIPrep release. If you're able to build the Docker image from a fork, you could manually switch from nonlinear T2* estimation to the log-linear version. Basically, you would just need to switch the order here: fmriprep/fmriprep/interfaces/multiecho.py Lines 64 to 73 in 32f7b06 Instead of |
|
Thanks for offering a solution while we wait for the fix, @tsalo. Do I need to include one of the flags I was using before ( I switched the order as you indicated, and tried to re-run fMRIPrep (without any of the above flags), and I got the broken process pool error again. I was testing with 3 individuals. ERROR |
|
Just wanted to send an update: We still have not been able to get the new version of fMRIPrep to work on more than 1 participant with multi echo data. |
|
I'm sorry for the delay in responding. If you look at the logs, do they show the t2smap command used? |
|
Yesterday I tried fmriprep 22.0.0rc3 on one subject (one session) and got the following error. It seems to me it's relevant but if it's not, please let me know and I delete my comment. Error message:Command I executed: Update: fmriprep 21.0.2 threw Here's the error: |
|
Did you ever get this working? I am having the same problem with t2smap but within a subject (many runs per subject) |
|
We successfully executed fmriprep 21.0.2. Here's my command: We haven't tried with newer versions. |
|
So the problem here is that @tsalo are there any extra coefficients or multiplicative factors that should be considered? At the very least, taking the full image set size and throwing in a factor of two will be a better estimate than 100M. |
|
I ran a local test and the relevant code (the bit that seemed to be failing) was maxing out at ~2 GB on a 288 MB file with three echoes. The proposed change in #2898 came out to 1.73 GB, so it might need a bit more? The function also uses |
|
Okay, a factor of 2.5 would get pretty close to 2gb. |
## Changes proposed in this pull request T2SMap was tagged with the default memory consumption (~100MB), when it should be based on the size of the input files. This implements #1100 in order to get a direct estimate of that, and then passes that along, with an additional factor based on the number of echos and a safety factor of 2. Fixes #1100. Fixes #2728. ## Documentation that should be reviewed <!-- Please summarize here the main changes to the documentation that the reviewers should be aware of. --> <!-- Welcome, new contributors! We ask you to read through the Contributing Guide: https://github.com/nipreps/fmriprep/blob/master/CONTRIBUTING.md These are guidelines intended to make communication easier by describing a consistent process, but don't worry if you don't get it everything exactly "right" on the first try. To boil it down, here are some highlights: 1) Consider starting a conversation in the issues list before submitting a pull request. The discussion might save you a lot of time coding. 2) Please use descriptive prefixes in your pull request title, such as "ENH:" for an enhancement or "FIX:" for a bug fix. (See the Contributing guide for the full set.) And consider adding a "WIP" tag for works-in-progress. 3) Any code you submit will be licensed under the same terms (Apache License 2.0) as the rest of fMRIPrep. 4) We invite every contributor to add themselves to the `.zenodo.json` file (https://github.com/nipreps/fmriprep/blob/master/.zenodo.json), which will result in your being listed as an author at the next release. Please add yourself as the next-to-last entry, just above Russ. A pull request is a conversation. We may ask you to make some changes before accepting your PR, and likewise, you should feel free to ask us any questions you have. -->
What happened?
We updated fMRIPrep to version 21.0.0. We have been using fMRIPrep for a few years now, and we have preprocessed multiple datasets with this pipeline. After making the upgrade, we wanted to re preprocess 2 of the datasets. We were able to re-preprocess the first no problem (NOT multi echo data). We went to preprocess the 2nd dataset - multiband/multiecho - and we can't preprocess more than 1 participant at a time. As I mentioned, these data have been preprocessed with fMRIPrep in the past no problem; I could run 10 at a time. (All work directories have been deleted, 'old' data have been moved out of the fMRIPrep path.) I am testing with 2-3 participants at a time, and have tried the following flags:
--low-mem
--mem-mb55000
--use-plugin plugin.yml (made a corresponding file - plugin: LegacyMultiProc
plugin_args: {maxtasksperchild: 1, memory_gb: 60, n_procs: 10, raise_insufficient: false}
Without the flags, and with the '--low-mem' flag, I get a 'broken process pull' and fMRIPrep doesn't finish. With the other 2 flags, I get a Node Name error: fmriprep_wf.single_subject_3156_wf.func_preproc_ses_1_task.... _ echo_1_wf.bold_t2smap_wf.t2nmap_node
(full error below)
I get this node error for each participant, different number of nodes.
If I run the participants separately, they run no problem.
What command did you use?
What version of fMRIPrep are you running?
21.0.0
How are you running fMRIPrep?
Singularity
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
No
Please copy and paste any relevant log output.
Additional information / screenshots
No response
The text was updated successfully, but these errors were encountered: