Updated documentation for running the template generation pipeline in Alliance Canada
#79
Conversation
| ``` | ||
| sbatch --time=24:00:00 --mem-per-cpu 4000 my_job.sh # will probably require batching several times, depending on number of subjects | ||
| sbatch --time=24:00:00 --mem-per-cpu 4000 template_pipeline.sh # will probably require batching several times, depending on number of subjects |
There was a problem hiding this comment.
what does "batching several times" mean?
There was a problem hiding this comment.
This part was written by Nadia. What I understood by that is that, the pipeline stops running in between and has to be re-run a few times. But this does add some confusion. Do you think I should remove it? It is implicitly understood.
There was a problem hiding this comment.
the pipeline stops running in between
and why is that?
There was a problem hiding this comment.
It is because of the out-of-memory issue. When I had discussed about this with Benjamin in January, he also mentioned -- I think it is because the resources from the core become unavailable.
In my case, when I ran it on CC, stopped the first time with the following issue:
slurmstepd: error: Detected 7 oom-kill event(s) in StepId=41187759.batch. Some of your processes may have been killed by the cgroup out-of-memory handler. and when I re-ran it, it has been running smoothly without any issues since yesterday evening. I just opened an issue on their repo to gain insights from the maintainers.
There was a problem hiding this comment.
ok— so add this info in the comment with a link to the issue for more clarity
Fixes #76, fixes #77
This
README.mdupdates some missing steps for setting up and running thegenerate_template.pyon Alliance Canada. Some specific details like setting up thenist_mni_pipelinesinside the template folder, creation of virtual environment etc were missing.To test this PR,
duke/temp/rohan/bids_datato the Alliance Canada cluster following step 2.a.Expected output
The pipeline would create a folder
model_n_allinside thetemplatefolder. For each and every iteration of the template generation process, a folder would be created inside themodel_n_allfolder.