Revision Request: Henriques, Rokem, Garyfallidis, St-Jean, Perterson, Correia 2/2

[RafaelNH]

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: Optimization of a free water elimination two-compartment model for diffusion tensor imaging
Author(s): Andrew R. Hoy, Cheng G. Koay, Steven R. Kecskemeti, Andrew L. Alexander
Journal (or Conference): NeuroImage
Year: 2014
DOI: 10.1016/j.neuroimage.2014.09.053
PDF: http://ac.els-cdn.com/S1053811914007952/1-s2.0-S1053811914007952-main.pdf?_tid=7b211902-b7b7-11e6-be78-00000aab0f6b&acdnat=1480591115_87d58e852819d91f683039ddc22f47d3**

Replication

Author(s): Rafael Neto Henriques, Ariel Rokem, Eleftherios Garyfallidis, Samuel St-Jean, Eric Thomas Peterson, Marta Morgado Correia
Repository: https://github.com/RafaelNH/ReScience-submission/tree/RNH-AR-EG-SSTJ-ETP-MMC-2016
PDF: https://github.com/RafaelNH/ReScience-submission/blob/RNH-AR-EG-SSTJ-ETP-MMC-2016/article/RNH_AR_EG_SSTJ_ETP_MMC-2016.pdf
Keywords: Diffusion MRI; Diffusion modeling; Diffusion tensor imaging; Partial volume; cerebrospinal fluid; free water elimination; partial volume effect
Language: English
Domain: Life Science

Results

• Article has been fully replicated
• Article has been partially replicated
• Article has not been replicated

Potential reviewers

---

EDITOR

• Editor acknowledgment (@pdebuyl) 2 December 2016
• Reviewer 1 (@delsuc) 5 December 2016
• Reviewer 2 (@soolijoo) 18 December 2016
• Review 1 decision [accept] 15 March 2017
• Review 2 decision [accept] 30 March 2017
• Editor decision [accept] 30 March 2017

[RafaelNH]

Dear @pdebuyl,
Due to its merge, I was not able to update PR #25. In this way, I decided to create a new PR to address the comments made by @delsuc and @soolijoo. Below you can find the responses to both reviewers.
Regards,
@RafaelNH

[RafaelNH]

Response to Reviewer 1

Thanks, @delsuc for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

1.1 General

We agreed that replicating the assessment of the optimal experimental set-up by varying the number of gradient and their intensities might be of interest. Therefore, the replication of figure 2 in the original article and the lower panels of figure 4 have been added to the manuscript. These new analyses evaluate both the optimal set of b-values and the number of b-value shells/directions. In our replication study, we decided to not incorporate Figure 3 of the original article, because this is only related to small adjustments to the two-shell acquisition scheme. We also didn’t incorporate the upper panels of Figure 4, since the larger errors associated to larger f-values are already shown in our article’s Figure 1. Instead, we decided to expand the analysis of the lower panels of Figure 4 to three diffusion measures rather than two, and for two FA levels. The relevant code can be found in the files run_simulations_2.py and run_simulations_3.py of code’s subfolder. Notebook version of these files can be found in the notebook subfolder (run_simulations_2.ipynb and run_simulations_3.ipynb).

1.2 Text

• The reason why eq 6 was reordered relative to the original article was added to the manuscript text (see the text right below eq.6): “Relative to the original article, the elements of $W$ have been re-ordered according to the order of the diffusion tensor elements used in Dipy.”
• Relatively to the cases of pure free water, we decided to give a deep look into the special case re-initializations proposed by Hoy et al. (2014). Based on the code in supplementary_notebook_3, we observed that the tissue’s mean diffusivity threshold of $1.5 \times 10 ^{-3} mm^{2}s^{-1}$ is more than adequate to identify voxels that contain only water. Therefore, we decided to use also this criterion instead of our initial implementation. With respect to the parameter re-initialization for this cases, some adjustments were done. These are justified in supplementary_notebook_1,ipynb.

1.2 Code

• A line of code was added in the notebooks to automatically access the functions directory.
• The dependency on Dipy was clarified in article’s subsection “Implementation dependencies”. In this subsection, we also provide some instructions, pointing to Dipy’s installation instructions on the (Dipy website). In addition, our repository includes a README.md file in the code y subfolder, that contains these instructions as well.
• Comments about the usage of different Dipy tools were added to all code files and notebooks.

1.3 misc remarks

• In run_data a comment was added to stress that 1.7 Gb will be downloaded.
• Typos in supplementary_notebook_1 and supplementary_notebook_2 were corrected

[RafaelNH]

Reviewer 2

Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

2.1 General

@soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study.

• In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript:

[8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A
diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245.

[9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white
matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133–
141.

[12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white
matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI
study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881.

2.2 Text

• The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference):

[11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn.
Reson. Med. 62.3 (Sept. 2009), pp. 717–730.

• Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).”

• We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (https://github.com/nipy/dipy/pull/835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this.

• Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations.

• The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises.

• A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014).

• We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55.

• Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “

2.3. Code

• Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this.

2.4. Minor points

• Detected Typos were amended
• Contrast of in vivo data was adjusted

[delsuc]

This version is an important rewrite of the original manuscript, and the text is very much improved.

thanks to there complete set-up, the authors have now been able to performs the simulation for the optimal acquisition conditions.
In addition, they have re-estimated of the best procedure for the analysis of voxels containing only free water, which have to be processed apart, and provide what seems to be an increase of the robustness of the method and a substantial improvement of the work itself.

Code is now easier to read, in particular thanks to the comment added, detailing the specific methods of the Dipy module.

All details have been fixed.

One remark, may be, for having the code available in the notebooks,
maybe the following syntax :

    import sys
    sys.path.append("../code")

is probably better than

    os.chdir('..//code')

as the authors do.

In conclusion I consider that this manuscript is ready to be published.

[pdebuyl]

Thanks @soolijoo for the clarification.

[pdebuyl]

Thank you @delsuc for the review.

@RafaelNH we are almost there I believe. Let us know when your final update is ready.

Regards, Pierre

[RafaelNH]

Thank you all for your comments.

@delsuc
In all notebooks, I changed “os.chdir('..//code')” to what you proposed.

@soolijoo

1. The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

2. I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes.

3. Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

4. Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:

(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

[pdebuyl]

Thank you @RafaelNH for this update.

@delsuc I consider acceptance on your side given your last message and the modification by @RafaelNH

@soolijoo can you confirm whether your last concerns were addressed?

[pdebuyl]

@soolijoo reminder

[soolijoo]

Responses are in bold below.

The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

Ok.

I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes.
Ok.

Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

Please remove the word "simulation" and replace it with the word "model" throughout.

Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:



(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

**This is a very indirect method of doing this -- my original comment included a mathworld link which allows evenly distributed points to be generated simply. Instead of sampling theta and phi unformaly, all that is required is to sample theta and cos(phi) uniformly. What you have done is fine, but dispersing points electrostatically is a more heavyweight operation.

Jones' goal is to construct reproducible sets of points for use as gradient direction sets. For this an electrostatic repulsion method is appropriate since it will lead to approximately even spacing for small numbers of points. You are making much larger sumbers of samples in order to get at the directional statistics of a problem and so a method which is assymtotically even will do.

There's no need to change this, the method is up to you.**

Overall
** I don't need to see another update, just make these changes and publish.**

[pdebuyl]

Hi @soolijoo thank you for this final review.

[RafaelNH]

Thanks @soolijoo for your final review.

@pdebuyl - I've just updated the manuscript with @soolijoo's final comment. The term multi-tensor simulation was replaced to multi-tensor model.

[pdebuyl]

Thanks everyone :-)

@RafaelNH I will start the publication/archival process in the coming days.

[pdebuyl]

@RafaelNH can you provide a few keywords for the article? You can find the keywords for previous ReScience articles here: http://rescience.github.io/read/ Typically they include the main programming language of the reproduction, the general field of research and a few more specific keywords.

[pdebuyl]

Also, can you confirm that the following are the proper last names of all authors? We use this for naming the archived version.

Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia

[RafaelNH]

Keywords: Neuroscience, python, diffusion-weighted imaging, diffusion modeling, diffusion tensor imaging, DTI, free water, partial volume, Cerebrospinal fluid, CSF.

I confirm that the last names are correct!

Many thanks!

[pdebuyl]

Hi @RafaelNH sorry for the delay, I proposed an update to the template that includes an article number, it should make citation a bit easier. Your article is ready for publication, you can check the final pdf here: https://github.com/ReScience-Archives/Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia-2017/tree/master/article

We don't typically do "proofs" in ReScience but why not :-) I am just waiting for your approval of the pdf to publish the final version. Please note that the day of publication is missing on purpose, I'll fill it in.

[RafaelNH]

Hi @pdebuyl! Is the page numeration correct?

[pdebuyl]

Yes. We use since now an article number. Pages are thus 2-1 to 2-10 for your article that is the second article in the volume 3, issue 1, of ReScience.

[RafaelNH]

Ok got it!
Is there any way to avoid having Table 2 split in 2 pages? Perhaps inserting this table after the second paragraph of Methods subsection 'Simulations 3' will solve the issue.
Apart from this minor, I approve the PDF.

[pdebuyl]

So?

[RafaelNH]

Hi @pdebuyl! Many thanks for the correction! I approve the PDF!

[pdebuyl]

EDITOR

This submission has been accepted for publication and will appear soon at http://rescience.github.io/read/

[pdebuyl]

All editorial steps done @rougier @khinsen

ReScience/rescience.github.io#31

ReScience/ReScience#49