Revision Request: Henriques, Rokem, Garyfallidis, St-Jean, Perterson, Correia 2/2 #26

Closed
wants to merge 25 commits into
from

Conversation

Projects
None yet
4 participants
@RafaelNH
Contributor

RafaelNH commented Feb 7, 2017

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: Optimization of a free water elimination two-compartment model for diffusion tensor imaging
Author(s): Andrew R. Hoy, Cheng G. Koay, Steven R. Kecskemeti, Andrew L. Alexander
Journal (or Conference): NeuroImage
Year: 2014
DOI: 10.1016/j.neuroimage.2014.09.053
PDF: http://ac.els-cdn.com/S1053811914007952/1-s2.0-S1053811914007952-main.pdf?_tid=7b211902-b7b7-11e6-be78-00000aab0f6b&acdnat=1480591115_87d58e852819d91f683039ddc22f47d3**

Replication

Author(s): Rafael Neto Henriques, Ariel Rokem, Eleftherios Garyfallidis, Samuel St-Jean, Eric Thomas Peterson, Marta Morgado Correia
Repository: https://github.com/RafaelNH/ReScience-submission/tree/RNH-AR-EG-SSTJ-ETP-MMC-2016
PDF: https://github.com/RafaelNH/ReScience-submission/blob/RNH-AR-EG-SSTJ-ETP-MMC-2016/article/RNH_AR_EG_SSTJ_ETP_MMC-2016.pdf
Keywords: Diffusion MRI; Diffusion modeling; Diffusion tensor imaging; Partial volume; cerebrospinal fluid; free water elimination; partial volume effect
Language: English
Domain: Life Science

Results

  • Article has been fully replicated
  • Article has been partially replicated
  • Article has not been replicated

Potential reviewers


EDITOR

  • Editor acknowledgment (@pdebuyl) 2 December 2016
  • Reviewer 1 (@delsuc) 5 December 2016
  • Reviewer 2 (@soolijoo) 18 December 2016
  • Review 1 decision [accept] 15 March 2017
  • Review 2 decision [accept] 30 March 2017
  • Editor decision [accept] 30 March 2017
@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Feb 7, 2017

Contributor

Dear @pdebuyl,
Due to its merge, I was not able to update PR #25. In this way, I decided to create a new PR to address the comments made by @delsuc and @soolijoo. Below you can find the responses to both reviewers.
Regards,
@RafaelNH

Contributor

RafaelNH commented Feb 7, 2017

Dear @pdebuyl,
Due to its merge, I was not able to update PR #25. In this way, I decided to create a new PR to address the comments made by @delsuc and @soolijoo. Below you can find the responses to both reviewers.
Regards,
@RafaelNH

@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Feb 7, 2017

Contributor

Response to Reviewer 1

Thanks, @delsuc for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

1.1 General

We agreed that replicating the assessment of the optimal experimental set-up by varying the number of gradient and their intensities might be of interest. Therefore, the replication of figure 2 in the original article and the lower panels of figure 4 have been added to the manuscript. These new analyses evaluate both the optimal set of b-values and the number of b-value shells/directions. In our replication study, we decided to not incorporate Figure 3 of the original article, because this is only related to small adjustments to the two-shell acquisition scheme. We also didn’t incorporate the upper panels of Figure 4, since the larger errors associated to larger f-values are already shown in our article’s Figure 1. Instead, we decided to expand the analysis of the lower panels of Figure 4 to three diffusion measures rather than two, and for two FA levels. The relevant code can be found in the files run_simulations_2.py and run_simulations_3.py of code’s subfolder. Notebook version of these files can be found in the notebook subfolder (run_simulations_2.ipynb and run_simulations_3.ipynb).

1.2 Text

  • The reason why eq 6 was reordered relative to the original article was added to the manuscript text (see the text right below eq.6): “Relative to the original article, the elements of $W$ have been re-ordered according to the order of the diffusion tensor elements used in Dipy.”
  • Relatively to the cases of pure free water, we decided to give a deep look into the special case re-initializations proposed by Hoy et al. (2014). Based on the code in supplementary_notebook_3, we observed that the tissue’s mean diffusivity threshold of $1.5 \times 10 ^{-3} mm^{2}s^{-1}$ is more than adequate to identify voxels that contain only water. Therefore, we decided to use also this criterion instead of our initial implementation. With respect to the parameter re-initialization for this cases, some adjustments were done. These are justified in supplementary_notebook_1,ipynb.

1.2 Code

  • A line of code was added in the notebooks to automatically access the functions directory.
  • The dependency on Dipy was clarified in article’s subsection “Implementation dependencies”. In this subsection, we also provide some instructions, pointing to Dipy’s installation instructions on the (Dipy website). In addition, our repository includes a README.md file in the code y subfolder, that contains these instructions as well.
  • Comments about the usage of different Dipy tools were added to all code files and notebooks.

1.3 misc remarks

  • In run_data a comment was added to stress that 1.7 Gb will be downloaded.
  • Typos in supplementary_notebook_1 and supplementary_notebook_2 were corrected
Contributor

RafaelNH commented Feb 7, 2017

Response to Reviewer 1

Thanks, @delsuc for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

1.1 General

We agreed that replicating the assessment of the optimal experimental set-up by varying the number of gradient and their intensities might be of interest. Therefore, the replication of figure 2 in the original article and the lower panels of figure 4 have been added to the manuscript. These new analyses evaluate both the optimal set of b-values and the number of b-value shells/directions. In our replication study, we decided to not incorporate Figure 3 of the original article, because this is only related to small adjustments to the two-shell acquisition scheme. We also didn’t incorporate the upper panels of Figure 4, since the larger errors associated to larger f-values are already shown in our article’s Figure 1. Instead, we decided to expand the analysis of the lower panels of Figure 4 to three diffusion measures rather than two, and for two FA levels. The relevant code can be found in the files run_simulations_2.py and run_simulations_3.py of code’s subfolder. Notebook version of these files can be found in the notebook subfolder (run_simulations_2.ipynb and run_simulations_3.ipynb).

1.2 Text

  • The reason why eq 6 was reordered relative to the original article was added to the manuscript text (see the text right below eq.6): “Relative to the original article, the elements of $W$ have been re-ordered according to the order of the diffusion tensor elements used in Dipy.”
  • Relatively to the cases of pure free water, we decided to give a deep look into the special case re-initializations proposed by Hoy et al. (2014). Based on the code in supplementary_notebook_3, we observed that the tissue’s mean diffusivity threshold of $1.5 \times 10 ^{-3} mm^{2}s^{-1}$ is more than adequate to identify voxels that contain only water. Therefore, we decided to use also this criterion instead of our initial implementation. With respect to the parameter re-initialization for this cases, some adjustments were done. These are justified in supplementary_notebook_1,ipynb.

1.2 Code

  • A line of code was added in the notebooks to automatically access the functions directory.
  • The dependency on Dipy was clarified in article’s subsection “Implementation dependencies”. In this subsection, we also provide some instructions, pointing to Dipy’s installation instructions on the (Dipy website). In addition, our repository includes a README.md file in the code y subfolder, that contains these instructions as well.
  • Comments about the usage of different Dipy tools were added to all code files and notebooks.

1.3 misc remarks

  • In run_data a comment was added to stress that 1.7 Gb will be downloaded.
  • Typos in supplementary_notebook_1 and supplementary_notebook_2 were corrected
@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Feb 7, 2017

Contributor

Reviewer 2

Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

2.1 General

@soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study.

  • In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript:

[8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A
diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245.

[9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white
matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133–
141.

[12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white
matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI
study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881.

2.2 Text

  • The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference):

[11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn.
Reson. Med. 62.3 (Sept. 2009), pp. 717–730.

  • Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).”

  • We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (nipy/dipy#835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this.

  • Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations.

  • The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises.

  • A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014).

  • We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55.

  • Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “

2.3. Code

  • Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this.

2.4. Minor points

  • Detected Typos were amended
  • Contrast of in vivo data was adjusted
Contributor

RafaelNH commented Feb 7, 2017

Reviewer 2

Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

2.1 General

@soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study.

  • In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript:

[8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A
diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245.

[9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white
matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133–
141.

[12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white
matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI
study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881.

2.2 Text

  • The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference):

[11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn.
Reson. Med. 62.3 (Sept. 2009), pp. 717–730.

  • Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).”

  • We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (nipy/dipy#835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this.

  • Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations.

  • The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises.

  • A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014).

  • We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55.

  • Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “

2.3. Code

  • Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this.

2.4. Minor points

  • Detected Typos were amended
  • Contrast of in vivo data was adjusted

@pdebuyl pdebuyl changed the title from Revision Request: Henriques, Rokem, Garyfallidis, St-Jean, Perterson, Correia to Revision Request: Henriques, Rokem, Garyfallidis, St-Jean, Perterson, Correia 2/2 Feb 7, 2017

@pdebuyl pdebuyl self-assigned this Feb 7, 2017

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Feb 7, 2017

Member

@RafaelNH Thanks for providing your reply to the referees.

@RafaelNH @arokem @delsuc @soolijoo The review process will continue here.

Member

pdebuyl commented Feb 7, 2017

@RafaelNH Thanks for providing your reply to the referees.

@RafaelNH @arokem @delsuc @soolijoo The review process will continue here.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Feb 13, 2017

Member

@delsuc @soolijoo can you acknowledge reception of the revision? This is to make sure that the notification did not get flushed down the inbox/notification. Thank you.

Member

pdebuyl commented Feb 13, 2017

@delsuc @soolijoo can you acknowledge reception of the revision? This is to make sure that the notification did not get flushed down the inbox/notification. Thank you.

@delsuc

This comment has been minimized.

Show comment
Hide comment
@delsuc

delsuc Feb 13, 2017

delsuc commented Feb 13, 2017

@soolijoo

This comment has been minimized.

Show comment
Hide comment
@soolijoo

soolijoo Feb 13, 2017

Contributor

I also acknowledge receipt.

Contributor

soolijoo commented Feb 13, 2017

I also acknowledge receipt.

@soolijoo

This comment has been minimized.

Show comment
Hide comment
@soolijoo

soolijoo Feb 23, 2017

Contributor

Thanks for these comments. I've added my response inline -- to make them easier to find, my additions are prefixed with **

Reviewer 2

Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

2.1 General

@soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study.

In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript:
[8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A
diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245.

[9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white
matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133–
141.

[12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white
matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI
study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881.

** These additions are welcome, but appear to be applications of the same two-tensor based postprocessing approach. My initial query was regarding why postprocessing was the right approach for handling free water rather than eliminating it in the acquisition. Please can you add a small section on why this is appropriate? A few sentences don't seem to be much to ask.

2.2 Text

The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference):
[11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn.
Reson. Med. 62.3 (Sept. 2009), pp. 717–730.

** see above.

Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).”

** OK, but it needs to be clear that tesnor data can't be inverted to probvode a unique signiture of these effects -- we see only their net effect.

We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (nipy/dipy#835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this.

** ok.

Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations.

** the link provided is to a definition of the Monte-Carlo method not Monte-Carlo simulation. The method approximates otherwise unfeasible integrals using random processes. a simulation is an application of the Monte-Carlo method to a given system and as such is a more specialised term. Some authors may indeed confuse these two points, but on my watch. Adding noise to an underlying analytical model is not a simulation, please amend this term.

The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises.

** ok.

A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014).

** ok.

We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55.

** ok.

Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “

** ok.

2.3. Code

Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this.

** my original point was not regarding diffusion gradient directions in the acquisition, it refers to the way in which you are generating random directions in your code. You pick theta and phi from a uniform distribution, which means all generated directions have a bias concentrating them around the poles of your coordinate system. The code needs to be amended such that theta and z are drawn from a uniform distributions, as per the procedure in the original link. Please amend this.

2.4. Minor points

Detected Typos were amended
Contrast of in vivo data was adjusted

** great.

Overall recommendation -- accept with minor corrections. Please address the remaining concerns highlighted in this review prior to publication.

Contributor

soolijoo commented Feb 23, 2017

Thanks for these comments. I've added my response inline -- to make them easier to find, my additions are prefixed with **

Reviewer 2

Thanks @soolijoo for your comments. Bellow, you can find a description of the changes done to address your comments for each of your review section.

2.1 General

@soolijoo pointed out some concerns regarding the theoretical background of the free water DTI (e.g. initial motivation of the use of this technique), the drawback of the simulations based on the same model, and the use of specific parameters such as the number of simulations repetitions and tested SNR levels. We added text to address these concerns, but we stress that we made these choices based primarily on our intention to replicate the work done by Hoy et al., (2014). This motivates the choice of SNR, and other choices of the simulation parameters. Further expansions of the original article go beyond the scope of this replication study.

In addition to the motivation given in the original manuscript, more clarifications on the motivation of the free water DTI model was added in the third paragraph of the manuscript. For further information, we point the reviewer to the following articles which were also added as references of our manuscript:
[8] C Metzler-Baddeley et al. “Frontotemporal connections in episodic memory and aging: A
diffusion MRI tractography study”. In: J. Neurosci 31.37 (Sept. 2011), pp. 13236–13245.

[9] L J O’Donnell and O Pasternak. “Does diffusion MRI tell us anything about the white
matter? An overview of methods and pitfalls.” In: Schizophr Res. 161.1 (Jan. 2015), pp. 133–
141.

[12] O Pasternak et al. “Hockey Concussion Education Project, Part 2. Microstructural white
matter alterations in acutely concussed ice hockey players: a longitudinal free-water MRI
study.” In: J Neurosurg. 120.4 (Apr. 2014), pp. 873–881.

** These additions are welcome, but appear to be applications of the same two-tensor based postprocessing approach. My initial query was regarding why postprocessing was the right approach for handling free water rather than eliminating it in the acquisition. Please can you add a small section on why this is appropriate? A few sentences don't seem to be much to ask.

2.2 Text

The advantages of using the free water DTI model relative to strategy to suppress free water using less conventional diffusion MRI sequences was added to the fourth paragraph of the manuscript introduction. For further information, we point reviewer to the following article (this article was also added as a reference):
[11] O Pasternak et al. “Free water elimination and mapping from diffusion MRI.” In: Magn.
Reson. Med. 62.3 (Sept. 2009), pp. 717–730.

** see above.

Page 1, Paragraph 2 was reworded to say "For example FA is sensitive to different microstructural properties (such as packing density of axons, and density of myelin in nerve fibers), as well to the degree of white matter coherence (i.e. the alignment of axons within a measurement voxel).”

** OK, but it needs to be clear that tesnor data can't be inverted to probvode a unique signiture of these effects -- we see only their net effect.

We also agree that C++ might provide more speed-efficient procedures. Indeed, our main objective here was to provide the first robust open-source reference implementation of fwDTI model. The code was written in Python, to facilitate both its readability (we closely adhere by the PEP8 standard to faciliate) and carefully implemented, including comprehensive test coverage, and revised using Github’s pull request mechanism (nipy/dipy#835). The choice of Python is also further motivated by the large community of potential users, through the Dipy community. While we agree that an even faster implementation could be designed in future work on this topic, this implementation already provides a 20-fold speedup relative to the timing reported by the authors of the original article. Nevertheless, we have rewritten the last paragraph of the introduction to clarify this.

** ok.

Some authors define Monte-Carlo simulations as any computational algorithms or method that rely on repeated random sampling to obtain numerical results (see for example the definition at Wolfram Mathworld )..Since this term was also adopted in the original article (Hoy et al., 2014), to keep consistency, we decide to continue refer to these simulations as Monte-Carlo simulations.

** the link provided is to a definition of the Monte-Carlo method not Monte-Carlo simulation. The method approximates otherwise unfeasible integrals using random processes. a simulation is an application of the Monte-Carlo method to a given system and as such is a more specialised term. Some authors may indeed confuse these two points, but on my watch. Adding noise to an underlying analytical model is not a simulation, please amend this term.

The justification for the chosen parameters to address the degeneracy of the free-water DTI model when voxel contain mostly water was added in the supplementary notebook 3. There we also address the frequency that this problem arises.

** ok.

A reference for the chosen free water diffusion value was added in the first paragraph of the methods section. This value is also consistent to what is normally assumed in previous diffusion MRI study including the original article (Hoy et al., 2014).

** ok.

We agree that an SNR of 20 is more typical in diffusion MRI experiments. We also don’t understand why the original article opted to use this SNR. Here we used this SNR so that figures are replicated in a consistent way relative to the original article. The evaluation of lower SNRs can be seen in the added figure 3. We also rerun Figure 1 for the SNR=20 - apart from larger noise bars, the general metrics bias didn’t change. We decided not to add this as an extra supplementary notebook just for the sake of reducing the number of supplementary notebook, however if interested in also inpecting this please run run_simulations_fig1.py after changing line 55.

** ok.

Second sentence of the first paragraph of the results section was reworded to: “As reported in the original article, no FA bias are observed for large FA ground truth values and free water volume fractions $f$ ranging around 0 to 0.7 … “

** ok.

2.3. Code

Diffusion gradient directions were already evenly sampled over the surface of a sphere using an electrostatic potential energy algorithm. This is a standard method used in the field to overcome the pitfall pointed out by the reviewer. Comments were added to the code to clarify this.

** my original point was not regarding diffusion gradient directions in the acquisition, it refers to the way in which you are generating random directions in your code. You pick theta and phi from a uniform distribution, which means all generated directions have a bias concentrating them around the poles of your coordinate system. The code needs to be amended such that theta and z are drawn from a uniform distributions, as per the procedure in the original link. Please amend this.

2.4. Minor points

Detected Typos were amended
Contrast of in vivo data was adjusted

** great.

Overall recommendation -- accept with minor corrections. Please address the remaining concerns highlighted in this review prior to publication.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 9, 2017

Member

Thank you @soolijoo for the report.

About the justification of the methods, do you have more specific concerns? The authors already commented on the existence of non-water-eliminated experimental data, for instance.

Member

pdebuyl commented Mar 9, 2017

Thank you @soolijoo for the report.

About the justification of the methods, do you have more specific concerns? The authors already commented on the existence of non-water-eliminated experimental data, for instance.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 9, 2017

Member

@delsuc reminder

Member

pdebuyl commented Mar 9, 2017

@delsuc reminder

@soolijoo

This comment has been minimized.

Show comment
Hide comment
@soolijoo

soolijoo Mar 10, 2017

Contributor

@pdebuyl I'm still not clear on what clinical or biophysical problem is caused by the the presence of free water without this explicit model. Why is it important to do this, and why is it worth the associated additional scan time and processing? Is there some pathology which cannot be imaged any other way, for example, or is there some widely-made inference which is biased by free water? The existance of free water in the signal is one thing, but why is this important in a given application?

I guess what I'm getting at is the killer reason why I should do this.

Contributor

soolijoo commented Mar 10, 2017

@pdebuyl I'm still not clear on what clinical or biophysical problem is caused by the the presence of free water without this explicit model. Why is it important to do this, and why is it worth the associated additional scan time and processing? Is there some pathology which cannot be imaged any other way, for example, or is there some widely-made inference which is biased by free water? The existance of free water in the signal is one thing, but why is this important in a given application?

I guess what I'm getting at is the killer reason why I should do this.

@delsuc

This comment has been minimized.

Show comment
Hide comment
@delsuc

delsuc Mar 10, 2017

@pdebuyl I'm working on it !

delsuc commented Mar 10, 2017

@pdebuyl I'm working on it !

@delsuc

This comment has been minimized.

Show comment
Hide comment
@delsuc

delsuc Mar 15, 2017

This version is an important rewrite of the original manuscript, and the text is very much improved.

thanks to there complete set-up, the authors have now been able to performs the simulation for the optimal acquisition conditions.
In addition, they have re-estimated of the best procedure for the analysis of voxels containing only free water, which have to be processed apart, and provide what seems to be an increase of the robustness of the method and a substantial improvement of the work itself.

Code is now easier to read, in particular thanks to the comment added, detailing the specific methods of the Dipy module.

All details have been fixed.

One remark, may be, for having the code available in the notebooks,
maybe the following syntax :

    import sys
    sys.path.append("../code")

is probably better than

    os.chdir('..//code')

as the authors do.

In conclusion I consider that this manuscript is ready to be published.

delsuc commented Mar 15, 2017

This version is an important rewrite of the original manuscript, and the text is very much improved.

thanks to there complete set-up, the authors have now been able to performs the simulation for the optimal acquisition conditions.
In addition, they have re-estimated of the best procedure for the analysis of voxels containing only free water, which have to be processed apart, and provide what seems to be an increase of the robustness of the method and a substantial improvement of the work itself.

Code is now easier to read, in particular thanks to the comment added, detailing the specific methods of the Dipy module.

All details have been fixed.

One remark, may be, for having the code available in the notebooks,
maybe the following syntax :

    import sys
    sys.path.append("../code")

is probably better than

    os.chdir('..//code')

as the authors do.

In conclusion I consider that this manuscript is ready to be published.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 17, 2017

Member

Thanks @soolijoo for the clarification.

Member

pdebuyl commented Mar 17, 2017

Thanks @soolijoo for the clarification.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 17, 2017

Member

Thank you @delsuc for the review.

@RafaelNH we are almost there I believe. Let us know when your final update is ready.

Regards, Pierre

Member

pdebuyl commented Mar 17, 2017

Thank you @delsuc for the review.

@RafaelNH we are almost there I believe. Let us know when your final update is ready.

Regards, Pierre

@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Mar 20, 2017

Contributor

Thank you all for your comments.

@delsuc
In all notebooks, I changed “os.chdir('..//code')” to what you proposed.

@soolijoo

  1. The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

  2. I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes.

  3. Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

  4. Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:

directions_code

(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

Contributor

RafaelNH commented Mar 20, 2017

Thank you all for your comments.

@delsuc
In all notebooks, I changed “os.chdir('..//code')” to what you proposed.

@soolijoo

  1. The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

  2. I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes.

  3. Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

  4. Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:

directions_code

(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 21, 2017

Member

Thank you @RafaelNH for this update.

@delsuc I consider acceptance on your side given your last message and the modification by @RafaelNH

@soolijoo can you confirm whether your last concerns were addressed?

Member

pdebuyl commented Mar 21, 2017

Thank you @RafaelNH for this update.

@delsuc I consider acceptance on your side given your last message and the modification by @RafaelNH

@soolijoo can you confirm whether your last concerns were addressed?

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 27, 2017

Member

@soolijoo reminder

Member

pdebuyl commented Mar 27, 2017

@soolijoo reminder

@soolijoo

This comment has been minimized.

Show comment
Hide comment
@soolijoo

soolijoo Mar 28, 2017

Contributor

Responses are in bold below.

The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

Ok.

I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes.
Ok.

Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

Please remove the word "simulation" and replace it with the word "model" throughout.

Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:

(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

**This is a very indirect method of doing this -- my original comment included a mathworld link which allows evenly distributed points to be generated simply. Instead of sampling theta and phi unformaly, all that is required is to sample theta and cos(phi) uniformly. What you have done is fine, but dispersing points electrostatically is a more heavyweight operation.

Jones' goal is to construct reproducible sets of points for use as gradient direction sets. For this an electrostatic repulsion method is appropriate since it will lead to approximately even spacing for small numbers of points. You are making much larger sumbers of samples in order to get at the directional statistics of a problem and so a method which is assymtotically even will do.

There's no need to change this, the method is up to you.**

Overall
** I don't need to see another update, just make these changes and publish.**

Contributor

soolijoo commented Mar 28, 2017

Responses are in bold below.

The paragraph 2, 3, 4 and 5 of the article's introduction was further edited to clarify the motivation of the use of the free water DTI model. The main objective of this model is to suppress the partial volume effects of CSF which diffusion coefficient equal to the expected free water value of 3.0e-3 mm2/s due to the absence of tissue barriers. In addition to this, recent studies have shown that measures of free water volume fractions can be also an indication extracellular changes. The use of this model was already shown promising results in removing partial volume confounds in ageing (Metzler-Baddeley, 2012) and to infer extracellular changes in the context of schizophrenia (Pasternak et al., 2012) and brain concussions (Pasternak et al., 2014).

Ok.

I also edited the 2nd paragraph of the article to clarify that DTI is not specific to concrete microstructural changes.
Ok.

Although this was the term used in the original article, I decided to replace the term Monte-Carlos simulations to multi-tensor simulations to avoid further confusions.

Please remove the word "simulation" and replace it with the word "model" throughout.

Regarding the sampling of random directions: we agree with the reviewer that a uniform sampling of directions does result in a bias, with a large concentration around the poles of the coordinate system. Indeed, this issue is well-known in diffusion MRI, and is already addressed in our implementation of the software: our software includes a correction of the initial uniform theta and phi angles. To correct this bias, we use the electrostatic potential energy algorithm (disperse_charges; Jones et al. 1999). For example, see the line of code highlighted in the following image:

(from https://github.com/RafaelNH/Free-water-elimination-DTI/blob/master/notebook/run_simulations_2.ipynb).

This was documented in the repository code and notebooks, but we have added a sentence in the methods mentioning the use of this procedure as well as a reference to the Jones et al. (1999) paper describing the algorithm and another explaining the issues in more detail (Jones et al., 2004). For a visual inspection of the performance of the repulsion algorithm please follow the following link: (http://nipy.org/dipy/examples_built/gradients_spheres.html#example-gradients-spheres).

**This is a very indirect method of doing this -- my original comment included a mathworld link which allows evenly distributed points to be generated simply. Instead of sampling theta and phi unformaly, all that is required is to sample theta and cos(phi) uniformly. What you have done is fine, but dispersing points electrostatically is a more heavyweight operation.

Jones' goal is to construct reproducible sets of points for use as gradient direction sets. For this an electrostatic repulsion method is appropriate since it will lead to approximately even spacing for small numbers of points. You are making much larger sumbers of samples in order to get at the directional statistics of a problem and so a method which is assymtotically even will do.

There's no need to change this, the method is up to you.**

Overall
** I don't need to see another update, just make these changes and publish.**

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 28, 2017

Member

Hi @soolijoo thank you for this final review.

Member

pdebuyl commented Mar 28, 2017

Hi @soolijoo thank you for this final review.

@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Mar 30, 2017

Contributor

Thanks @soolijoo for your final review.

@pdebuyl - I've just updated the manuscript with @soolijoo's final comment. The term multi-tensor simulation was replaced to multi-tensor model.

Contributor

RafaelNH commented Mar 30, 2017

Thanks @soolijoo for your final review.

@pdebuyl - I've just updated the manuscript with @soolijoo's final comment. The term multi-tensor simulation was replaced to multi-tensor model.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 30, 2017

Member

Thanks everyone :-)

@RafaelNH I will start the publication/archival process in the coming days.

Member

pdebuyl commented Mar 30, 2017

Thanks everyone :-)

@RafaelNH I will start the publication/archival process in the coming days.

@ReScience ReScience locked and limited conversation to collaborators Mar 30, 2017

@ReScience ReScience unlocked this conversation Mar 31, 2017

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 31, 2017

Member

@RafaelNH can you provide a few keywords for the article? You can find the keywords for previous ReScience articles here: http://rescience.github.io/read/ Typically they include the main programming language of the reproduction, the general field of research and a few more specific keywords.

Member

pdebuyl commented Mar 31, 2017

@RafaelNH can you provide a few keywords for the article? You can find the keywords for previous ReScience articles here: http://rescience.github.io/read/ Typically they include the main programming language of the reproduction, the general field of research and a few more specific keywords.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Mar 31, 2017

Member

Also, can you confirm that the following are the proper last names of all authors? We use this for naming the archived version.

Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia
Member

pdebuyl commented Mar 31, 2017

Also, can you confirm that the following are the proper last names of all authors? We use this for naming the archived version.

Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia
@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Mar 31, 2017

Contributor

Keywords: Neuroscience, python, diffusion-weighted imaging, diffusion modeling, diffusion tensor imaging, DTI, free water, partial volume, Cerebrospinal fluid, CSF.

I confirm that the last names are correct!

Many thanks!

Contributor

RafaelNH commented Mar 31, 2017

Keywords: Neuroscience, python, diffusion-weighted imaging, diffusion modeling, diffusion tensor imaging, DTI, free water, partial volume, Cerebrospinal fluid, CSF.

I confirm that the last names are correct!

Many thanks!

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Apr 5, 2017

Member

Hi @RafaelNH sorry for the delay, I proposed an update to the template that includes an article number, it should make citation a bit easier. Your article is ready for publication, you can check the final pdf here: https://github.com/ReScience-Archives/Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia-2017/tree/master/article

We don't typically do "proofs" in ReScience but why not :-) I am just waiting for your approval of the pdf to publish the final version. Please note that the day of publication is missing on purpose, I'll fill it in.

Member

pdebuyl commented Apr 5, 2017

Hi @RafaelNH sorry for the delay, I proposed an update to the template that includes an article number, it should make citation a bit easier. Your article is ready for publication, you can check the final pdf here: https://github.com/ReScience-Archives/Henriques-Rokem-Garyfallidis-St-Jean-Peterson-Correia-2017/tree/master/article

We don't typically do "proofs" in ReScience but why not :-) I am just waiting for your approval of the pdf to publish the final version. Please note that the day of publication is missing on purpose, I'll fill it in.

@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Apr 5, 2017

Contributor

Hi @pdebuyl! Is the page numeration correct?

Contributor

RafaelNH commented Apr 5, 2017

Hi @pdebuyl! Is the page numeration correct?

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Apr 5, 2017

Member

Yes. We use since now an article number. Pages are thus 2-1 to 2-10 for your article that is the second article in the volume 3, issue 1, of ReScience.

Member

pdebuyl commented Apr 5, 2017

Yes. We use since now an article number. Pages are thus 2-1 to 2-10 for your article that is the second article in the volume 3, issue 1, of ReScience.

@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Apr 5, 2017

Contributor

Ok got it!
Is there any way to avoid having Table 2 split in 2 pages? Perhaps inserting this table after the second paragraph of Methods subsection 'Simulations 3' will solve the issue.
Apart from this minor, I approve the PDF.

Contributor

RafaelNH commented Apr 5, 2017

Ok got it!
Is there any way to avoid having Table 2 split in 2 pages? Perhaps inserting this table after the second paragraph of Methods subsection 'Simulations 3' will solve the issue.
Apart from this minor, I approve the PDF.

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Apr 5, 2017

Member

So?

Member

pdebuyl commented Apr 5, 2017

So?

@RafaelNH

This comment has been minimized.

Show comment
Hide comment
@RafaelNH

RafaelNH Apr 5, 2017

Contributor

Hi @pdebuyl! Many thanks for the correction! I approve the PDF!

Contributor

RafaelNH commented Apr 5, 2017

Hi @pdebuyl! Many thanks for the correction! I approve the PDF!

@pdebuyl

This comment has been minimized.

Show comment
Hide comment
@pdebuyl

pdebuyl Apr 5, 2017

Member

EDITOR

This submission has been accepted for publication and will appear soon at http://rescience.github.io/read/

DOI

Member

pdebuyl commented Apr 5, 2017

EDITOR

This submission has been accepted for publication and will appear soon at http://rescience.github.io/read/

DOI

@pdebuyl

This comment has been minimized.

Show comment
Hide comment

@pdebuyl pdebuyl closed this Apr 5, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment