-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updated documentation for running the template generation pipeline
in Alliance Canada
#79
Conversation
``` | ||
sbatch --time=24:00:00 --mem-per-cpu 4000 my_job.sh # will probably require batching several times, depending on number of subjects | ||
sbatch --time=24:00:00 --mem-per-cpu 4000 template_pipeline.sh # will probably require batching several times, depending on number of subjects |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what does "batching several times" mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This part was written by Nadia. What I understood by that is that, the pipeline stops running in between and has to be re-run a few times. But this does add some confusion. Do you think I should remove it? It is implicitly understood.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the pipeline stops running in between
and why is that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is because of the out-of-memory issue. When I had discussed about this with Benjamin in January, he also mentioned -- I think it is because the resources from the core become unavailable.
In my case, when I ran it on CC, stopped the first time with the following issue:
slurmstepd: error: Detected 7 oom-kill event(s) in StepId=41187759.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.
and when I re-ran it, it has been running smoothly without any issues since yesterday evening. I just opened an issue on their repo to gain insights from the maintainers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok— so add this info in the comment with a link to the issue for more clarity
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Fixes #76, fixes #77
This
README.md
updates some missing steps for setting up and running thegenerate_template.py
on Alliance Canada. Some specific details like setting up thenist_mni_pipelines
inside the template folder, creation of virtual environment etc were missing.To test this PR,
duke/temp/rohan/bids_data
to the Alliance Canada cluster following step 2.a.Expected output
The pipeline would create a folder
model_n_all
inside thetemplate
folder. For each and every iteration of the template generation process, a folder would be created inside themodel_n_all
folder.