Aperture Review

[ccraddock]

I couldn't figure out how to post my review on the OSF Preprint page so I am putting it here. Hopefully that will not cause any issues.

The article “BrainIAK: The Brain Imaging Analysis Kit” describes a software package for performing several different advanced analyses of functional neuroimaging data and resources for learning how to use it. Overall the paper is well written and provides detailed and informative descriptions of each of the included methods. The notebooks are well done and provide helpful demos to help users more quickly learn how to use the system. By optimizing the software to run efficiently on HPC systems, the developers have further empowered users to scale up their analysis. Overall, nicely done!

I enjoyed the section on optimization and was glad to see that a notebook with scaling advice is provided. I would prefer to see some of this information in the paper, perhaps each method could be accompanied by a brief section that provides this advice? In particular details about what dimension of the data is most crucial for an algorithm, and how to set bounds to avoid overwhelming the system.

I couldn’t find information about how Brainiak handles scheduling itself to run on clusters. What types of clusters and cluster software does it support? Or does it leave this detail to the user?

It would be nice to include some more information about how the software is developed. It appears to be primarily writing in Python with at least some of the more computationally intensive code written in C++. Also, please provide some information about how testing is performed at various levels. Are their datasets that the users can use to verify or benchmark their installations? Are their CI/CD systems that can automatically test pull requests?

It is nice to see the section on how to contribute to Brainiak. It would be useful to also include code standard requirements, any form of autolinting that will be performed on PRs, and details about how PRs are reviewed prior to merging.

In section 2.5 it might be useful to mention the similarity of the implemented methods to methods developed for dynamic functional connectivity analyses e.g., PMID 19629982 and PMC3920766. Is it possible that the described HMM approach could be used for dynamic FC with some modification?

In the introduction to real time fMRI section, the statement “Existing software frameworks have helped researchers to better implement RT-fMRI studies.” appears to be missing citations.

The optimization section mentions searchlight analysis, but I couldn’t find it mentioned in the earlier descriptions of methods. Are all of the methods in the toolbox described here? If not, it would be good to include a table of all of the methods included in the toolbox along with citations.

[manojneuro]

@ccraddock: Thank you for your review comments on the manuscript. We will update the manuscript with your suggestions and get back with an revised version.