Running in Parallel

[ValGitHu]

Hi,
I am trying to run the morphology pipeline in parallel using the GUI and I am getting the following error ‘output_database’ not defined.
Everything works well if I run the same processes in serie. Any suggestion?
Thank you,
V

[denisri]

output_database is only used in import processes and capsul variants of the pipeline. Are you talking about the regular Morphologist 2021 pipeline, or while importing data, or anything else ?
The Import T1 MRI process is one of the few which need databasing during its execution, which is disabled in parallel execution, thus this specific process cannot run in a parallel execution, but Morphologist should.

[ValGitHu]

You are right, I am importing the freesurfer outputs. Is there any option to run this step in parallel ? Maybe using python scripts ? thank you!

[denisri]

Ah OK then I understand... :)
Well yes, the T1 import process is using the database attributes during execution, which is somewhat wrong and can probably be avoided. Moreover the output_database parameter is mandatory, but is only a trick to make parameters links work, and is not used during execution, so it should not be mandatory. I'll see if I can fix a few things.

[denisri]

I could fix the import process quite easily, but the freesurfer import process has a more important problem: it is running a (partial) Morphologist process and gives it only a few filenames, letting the completion system determine the others. But the completion system needs databasing, thus the process doesn't work in parallel execution.
So in the current state, the FS import process cannot run in parallel.
To fix it, technically, there are 2 ways:

• allow databasing during parallel execution: we could imagine that some processes could declare that they require databasing, and for them databasing would be enabled (using the --enabledb option of the axon-runprocess command used in jobs), but 1) it would require API changes in processes and the infrastructure, 2) databases could be accessed from several computing nodes simultaneously, and the sqlite database locking system doesn't work well, especially on network filesystems, which is the main reason why we have disabled it. It could perhaps work on a local filesystem (thus on a local computer), but wouldn't be very clean...
• completely re-write the freesufer/morphologist import process, and make it a pipeline containing (and exposing) all Morphologist pipeline structure and parameters. This way all needed parameters would be completed and set before running. This would require some work, which we don't have time for right now...

So for now the short answer is: no it's not possible to run this specific process in parallel.

[wsnyder4]

I was able to parallelize this process but in a bit of an unconventional way to meet the needs of a cluster I am computing on. Specifically, my cluster needed all data/scripts to be copied to a directory created for each job (one job per subject) submitted, populated upon job submission.

If you use the singularity distribution and then use axon-runprocess to run a *.bvproc file from the command line (i.e., the process for importing freesurfer files, which you can specify inputs/outputs in and then save) without using any fancier singularity commands to bind directories, then the process will automatically access databases associated with the brainvisa installation specified by the PATH environmental variable.

So, you could technically just make a new copy of brainvisa in a new directory for each subject, and then add that install to the top of the the path with export PATH={subject_directory}/brainvisa-5.0.3/bin:${PATH} . You can delete the copy of the install when you're done running the subject.

This should work because I believe that if you create a shell script for each subject, then that shell script should only have the PATH updated within its own processing (avoiding cross-talk between PATH variables). This is because shell scripts inherit the PATH from its parent but do not modify its parent, and so your path will end up including your original brainvisa install and this copied one, but the copied one gets picked up first with the export command I showed.

You must create the database in your original brainvisa install location, such as brainvisa-5.0.3/home/database (and make sure you have checked in the GUI that this database is selected in. preferences). When you copy the whole brainvisa install into a subject directory, then axon-runprocess (with the updated path) will access the database in this new location.

This a kind of hack (and may take some playing around with) and probably not optimal for most, so I wouldn't necessarily recommend it but could be useful if your planning to run many subjects at once on a cluster (and can spare the extra GBs for copying / deleting the brainvisa copies you make for each subject).

[denisri]

I could make an implementation of solution 2) work.
You need to install brainvisa in read-write mode (as explained in #85), then update the files python/brainvisa/axon/runprocess.py, brainvisa/toolboxes/data management/processes/import/t1mri/ImportT1MRI.py and brainvisa/toolboxes/freesurfer/processes/Import/Import_FROM_FreeSurfer_TO_Morpho.py in the R/W install from the updated ones found in:

• https://brainvisa.info/download/updates/5.0.3/python/brainvisa/axon/runprocess.py
• https://brainvisa.info/download/updates/5.0.3/brainvisa/toolboxes/data%20management/processes/import/t1mri/ImportT1MRI.py
• https://brainvisa.info/download/updates/5.0.3/brainvisa/toolboxes/freesurfer/processes/Import/Import_FROM_FreeSurfer_TO_Morpho.py

The update will allow the process to force enabling databasing during execution, thus completion will happen.