# Revision Request: Henriques, Rokem, Garyfallidis, St-Jean, Perterson, Correia 2/2 #26

Closed
wants to merge 25 commits into
from

## Conversation

Projects
None yet
4 participants
Contributor

### RafaelNH commented Feb 7, 2017 • edited by pdebuyl Edited 1 time pdebuyl edited Mar 30, 2017 (most recent)

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

### Original article

Title: Optimization of a free water elimination two-compartment model for diffusion tensor imaging
Author(s): Andrew R. Hoy, Cheng G. Koay, Steven R. Kecskemeti, Andrew L. Alexander
Journal (or Conference): NeuroImage
Year: 2014
DOI: 10.1016/j.neuroimage.2014.09.053
PDF: http://ac.els-cdn.com/S1053811914007952/1-s2.0-S1053811914007952-main.pdf?_tid=7b211902-b7b7-11e6-be78-00000aab0f6b&acdnat=1480591115_87d58e852819d91f683039ddc22f47d3**

### Replication

Author(s): Rafael Neto Henriques, Ariel Rokem, Eleftherios Garyfallidis, Samuel St-Jean, Eric Thomas Peterson, Marta Morgado Correia
Repository: https://github.com/RafaelNH/ReScience-submission/tree/RNH-AR-EG-SSTJ-ETP-MMC-2016
PDF: https://github.com/RafaelNH/ReScience-submission/blob/RNH-AR-EG-SSTJ-ETP-MMC-2016/article/RNH_AR_EG_SSTJ_ETP_MMC-2016.pdf
Keywords: Diffusion MRI; Diffusion modeling; Diffusion tensor imaging; Partial volume; cerebrospinal fluid; free water elimination; partial volume effect
Language: English
Domain: Life Science

### Results

• Article has been fully replicated
• Article has been partially replicated
• Article has not been replicated

### Potential reviewers

EDITOR

• Editor acknowledgment (@pdebuyl) 2 December 2016
• Reviewer 1 (@delsuc) 5 December 2016
• Reviewer 2 (@soolijoo) 18 December 2016
• Review 1 decision [accept] 15 March 2017
• Review 2 decision [accept] 30 March 2017
• Editor decision [accept] 30 March 2017

### RafaelNH added some commits Nov 28, 2016

 Add/updates file to build the article's PDF 
 b5c741b 
 Add code files 
 0492be4 
 Change README file with the description of code files and requirement… 
…s to process them
 ef1ab73 
 Add notebooks and edit the file README with notebook descriptions 
 9d7fb09 
 DOC: Some small adjustments in manuscript 
 388f175 
 Update code license and add license for notebooks 
 16af9cc 
 Add to notebooks: from __future__ import print_function 
 9d3d8eb 
 DOC: First article changes: 
1) Add more information about methods motivation
2) Add paragraph with advantages of the fwDTI model compared to nulling free water in the acquisition
3) Explain why eq 6 was reordered
 583df52 
 RF: Adjustments in the initialization of parameters to suppress artif… 
…acts in voxels containing only free water
 a994906 
 update bibliography.bib 
 7c13514 
 Update article's subsections 'Removing problematic estimates' and 'Im… 
…plementation dependencies
 4550ccd 
 Add more details of Dipy's installation in the README file of code su… 
…bfolder
 6f81537 
 Add new simulations in method description 
 fbb9dcc 
 Add new simulations in text 
 8a38651 
 Add new figures 
 faefa63 
 Add updated version of code 
 6c7678a 
 Add update version of article and its figures 
 db88748 
 Add updated version of figure of in vivo data 
 63c6ebc 
 Add right supplementary notebooks order 
 353938d 
 Correct typos in figure 2 lables 
 db4e89a 
 Proof read supplementary notebook 1 
 4b70961 
 Last proof read of article 
 f1fb0d8 
Contributor

### RafaelNH commented Feb 7, 2017

 Dear @pdebuyl, Due to its merge, I was not able to update PR #25. In this way, I decided to create a new PR to address the comments made by @delsuc and @soolijoo. Below you can find the responses to both reviewers. Regards, @RafaelNH
Contributor

### RafaelNH commented Feb 7, 2017 • edited Edited 1 time RafaelNH edited Feb 7, 2017 (most recent)

 Response to Reviewer 1 Thanks, @delsuc for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section. 1.1 General We agreed that replicating the assessment of the optimal experimental set-up by varying the number of gradient and their intensities might be of interest. Therefore, the replication of figure 2 in the original article and the lower panels of figure 4 have been added to the manuscript. These new analyses evaluate both the optimal set of b-values and the number of b-value shells/directions. In our replication study, we decided to not incorporate Figure 3 of the original article, because this is only related to small adjustments to the two-shell acquisition scheme. We also didn’t incorporate the upper panels of Figure 4, since the larger errors associated to larger f-values are already shown in our article’s Figure 1. Instead, we decided to expand the analysis of the lower panels of Figure 4 to three diffusion measures rather than two, and for two FA levels. The relevant code can be found in the files run_simulations_2.py and run_simulations_3.py of code’s subfolder. Notebook version of these files can be found in the notebook subfolder (run_simulations_2.ipynb and run_simulations_3.ipynb). 1.2 Text The reason why eq 6 was reordered relative to the original article was added to the manuscript text (see the text right below eq.6): “Relative to the original article, the elements of $W$ have been re-ordered according to the order of the diffusion tensor elements used in Dipy.” Relatively to the cases of pure free water, we decided to give a deep look into the special case re-initializations proposed by Hoy et al. (2014). Based on the code in supplementary_notebook_3, we observed that the tissue’s mean diffusivity threshold of $1.5 \times 10 ^{-3} mm^{2}s^{-1}$ is more than adequate to identify voxels that contain only water. Therefore, we decided to use also this criterion instead of our initial implementation. With respect to the parameter re-initialization for this cases, some adjustments were done. These are justified in supplementary_notebook_1,ipynb. 1.2 Code A line of code was added in the notebooks to automatically access the functions directory. The dependency on Dipy was clarified in article’s subsection “Implementation dependencies”. In this subsection, we also provide some instructions, pointing to Dipy’s installation instructions on the (Dipy website). In addition, our repository includes a README.md file in the code y subfolder, that contains these instructions as well. Comments about the usage of different Dipy tools were added to all code files and notebooks. 1.3 misc remarks In run_data a comment was added to stress that 1.7 Gb will be downloaded. Typos in supplementary_notebook_1 and supplementary_notebook_2 were corrected
Contributor

### RafaelNH commented Feb 7, 2017

 Reviewer 2 Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section. 2.1 General @soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study. In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript: [8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245. [9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133– 141. [12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881. 2.2 Text The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference): [11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn. Reson. Med. 62.3 (Sept. 2009), pp. 717–730. Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).” We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (nipy/dipy#835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this. Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations. The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises. A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014). We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55. Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “ 2.3. Code Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this. 2.4. Minor points Detected Typos were amended Contrast of in vivo data was adjusted

Merged

Member

### pdebuyl commented Feb 7, 2017

 @RafaelNH Thanks for providing your reply to the referees. @RafaelNH @arokem @delsuc @soolijoo The review process will continue here.
Member

### pdebuyl commented Feb 13, 2017

 @delsuc @soolijoo can you acknowledge reception of the revision? This is to make sure that the notification did not get flushed down the inbox/notification. Thank you.

### delsuc commented Feb 13, 2017

 I acknowledge the reception, I was in congress last week, and I’ll be in vacation (off-grid :-) next week, I’ll try to do my best rapidly Marc-André … On 13 Feb 2017, at 11:25, Pierre de Buyl ***@***.***> wrote: @delsuc @soolijoo can you acknowledge reception of the revision? This is to make sure that the notification did not get flushed down the inbox/notification. Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
Contributor

### soolijoo commented Feb 13, 2017

 I also acknowledge receipt.
 Correct minor typos 
 77e665b 
Contributor

### soolijoo commented Feb 23, 2017

 Thanks for these comments. I've added my response inline -- to make them easier to find, my additions are prefixed with ** Reviewer 2 Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section. 2.1 General @soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study. In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript: [8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245. [9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133– 141. [12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881. ** These additions are welcome, but appear to be applications of the same two-tensor based postprocessing approach. My initial query was regarding why postprocessing was the right approach for handling free water rather than eliminating it in the acquisition. Please can you add a small section on why this is appropriate? A few sentences don't seem to be much to ask. 2.2 Text The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference): [11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn. Reson. Med. 62.3 (Sept. 2009), pp. 717–730. ** see above. Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).” ** OK, but it needs to be clear that tesnor data can't be inverted to probvode a unique signiture of these effects -- we see only their net effect. We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (nipy/dipy#835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this. ** ok. Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations. ** the link provided is to a definition of the Monte-Carlo method not Monte-Carlo simulation. The method approximates otherwise unfeasible integrals using random processes. a simulation is an application of the Monte-Carlo method to a given system and as such is a more specialised term. Some authors may indeed confuse these two points, but on my watch. Adding noise to an underlying analytical model is not a simulation, please amend this term. The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises. ** ok. A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014). ** ok. We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55. ** ok. Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “ ** ok. 2.3. Code Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this. ** my original point was not regarding diffusion gradient directions in the acquisition, it refers to the way in which you are generating random directions in your code. You pick theta and phi from a uniform distribution, which means all generated directions have a bias concentrating them around the poles of your coordinate system. The code needs to be amended such that theta and z are drawn from a uniform distributions, as per the procedure in the original link. Please amend this. 2.4. Minor points Detected Typos were amended Contrast of in vivo data was adjusted ** great. Overall recommendation -- accept with minor corrections. Please address the remaining concerns highlighted in this review prior to publication.
Member

### pdebuyl commented Mar 9, 2017

 Thank you @soolijoo for the report. About the justification of the methods, do you have more specific concerns? The authors already commented on the existence of non-water-eliminated experimental data, for instance.
Member

### pdebuyl commented Mar 9, 2017

 @delsuc reminder
Contributor

### soolijoo commented Mar 10, 2017

 @pdebuyl I'm still not clear on what clinical or biophysical problem is caused by the the presence of free water without this explicit model. Why is it important to do this, and why is it worth the associated additional scan time and processing? Is there some pathology which cannot be imaged any other way, for example, or is there some widely-made inference which is biased by free water? The existance of free water in the signal is one thing, but why is this important in a given application? I guess what I'm getting at is the killer reason why I should do this.

### delsuc commented Mar 10, 2017

 @pdebuyl I'm working on it !

### delsuc commented Mar 15, 2017

 This version is an important rewrite of the original manuscript, and the text is very much improved. thanks to there complete set-up, the authors have now been able to performs the simulation for the optimal acquisition conditions. In addition, they have re-estimated of the best procedure for the analysis of voxels containing only free water, which have to be processed apart, and provide what seems to be an increase of the robustness of the method and a substantial improvement of the work itself. Code is now easier to read, in particular thanks to the comment added, detailing the specific methods of the Dipy module. All details have been fixed. One remark, may be, for having the code available in the notebooks, maybe the following syntax :  import sys sys.path.append("../code")  is probably better than  os.chdir('..//code')  as the authors do. In conclusion I consider that this manuscript is ready to be published.
Member

### pdebuyl commented Mar 17, 2017

 Thanks @soolijoo for the clarification.
Member

### pdebuyl commented Mar 17, 2017

 Thank you @delsuc for the review. @RafaelNH we are almost there I believe. Let us know when your final update is ready. Regards, Pierre
 Revision 2 
 20f821e 
Contributor

### RafaelNH commented Mar 20, 2017

 Thank you all for your comments. @delsuc In all notebooks, I changed “os.chdir('..//code')” to what you proposed. @soolijoo The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014). I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes. Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions. Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image: This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).
Member

### pdebuyl commented Mar 21, 2017

 Thank you @RafaelNH for this update. @delsuc I consider acceptance on your side given your last message and the modification by @RafaelNH @soolijoo can you confirm whether your last concerns were addressed?
Member

### pdebuyl commented Mar 27, 2017

 @soolijoo reminder
Contributor

Member

### pdebuyl commented Mar 28, 2017

 Hi @soolijoo thank you for this final review.
 DOC: multi-tensor simulations -> multi-tensor models 
 8dad23f 
Contributor

### RafaelNH commented Mar 30, 2017

 Thanks @soolijoo for your final review. @pdebuyl - I've just updated the manuscript with @soolijoo's final comment. The term multi-tensor simulation was replaced to multi-tensor model.
Member

### pdebuyl commented Mar 30, 2017

 Thanks everyone :-) @RafaelNH I will start the publication/archival process in the coming days.

Member

### pdebuyl commented Mar 31, 2017

 @RafaelNH can you provide a few keywords for the article? You can find the keywords for previous ReScience articles here: http://rescience.github.io/read/ Typically they include the main programming language of the reproduction, the general field of research and a few more specific keywords.
Member

### pdebuyl commented Mar 31, 2017

 Also, can you confirm that the following are the proper last names of all authors? We use this for naming the archived version. Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia 
Contributor

### RafaelNH commented Mar 31, 2017

 Keywords: Neuroscience, python, diffusion-weighted imaging, diffusion modeling, diffusion tensor imaging, DTI, free water, partial volume, Cerebrospinal fluid, CSF. I confirm that the last names are correct! Many thanks!

Closed

Open

Member

### pdebuyl commented Apr 5, 2017

 Hi @RafaelNH sorry for the delay, I proposed an update to the template that includes an article number, it should make citation a bit easier. Your article is ready for publication, you can check the final pdf here: https://github.com/ReScience-Archives/Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia-2017/tree/master/article We don't typically do "proofs" in ReScience but why not :-) I am just waiting for your approval of the pdf to publish the final version. Please note that the day of publication is missing on purpose, I'll fill it in.
Contributor

### RafaelNH commented Apr 5, 2017

 Hi @pdebuyl! Is the page numeration correct?
Member

### pdebuyl commented Apr 5, 2017

 Yes. We use since now an article number. Pages are thus 2-1 to 2-10 for your article that is the second article in the volume 3, issue 1, of ReScience.
Contributor

### RafaelNH commented Apr 5, 2017

 Ok got it! Is there any way to avoid having Table 2 split in 2 pages? Perhaps inserting this table after the second paragraph of Methods subsection 'Simulations 3' will solve the issue. Apart from this minor, I approve the PDF.
Member

 So?
Contributor

### RafaelNH commented Apr 5, 2017

 Hi @pdebuyl! Many thanks for the correction! I approve the PDF!
Member

### pdebuyl commented Apr 5, 2017

 EDITOR This submission has been accepted for publication and will appear soon at http://rescience.github.io/read/