Review Request: Detorakis #17

Closed
wants to merge 11 commits into
from

Conversation

Projects
None yet
7 participants
@gdetor

gdetor commented May 6, 2016

Dear @ReScience/editors,

I request a review for the reproduction of the following paper:

  • Multiple dynamical modes of thalamic relay neurons: rhythmic bursting and
    intermittent phase-locking, Wang, X-J, Neuroscience, 59(1), pg. 21–31, 1994.

I believe the original results have been faithfully reproduced as explained in the accompanying article.

The repository lives @ https://github.com/gdetor/ReScience-submission/tree/detorakis

Best regards,
Georgios Detorakis


EDITOR

  • Editor acknowledgment (@otizonaizit) May 9, 2016
  • Reviewer 1 (@heplesser) May 10, 2016
  • Reviewer 2 (@apdavison) May 9, 2016
  • Review 1 decision [accept] Aug 9, 2016
  • Review 2 decision [accept] Aug 28, 2016
  • Editor decision [accept] Aug 29, 2016
@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier May 6, 2016

Member

@gdetor Thanks, an editor will be soon assigned.
@otizonaizit Can you handle this request (I've a conflict of interest) ?

Member

rougier commented May 6, 2016

@gdetor Thanks, an editor will be soon assigned.
@otizonaizit Can you handle this request (I've a conflict of interest) ?

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit May 9, 2016

Member

EDITOR
@rougier: yep, I am on it.

Member

otizonaizit commented May 9, 2016

EDITOR
@rougier: yep, I am on it.

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit May 9, 2016

Member

EDITOR
@heplesser: Can you be REVIEWER 1 for this?
@apdavison: Can you be REVIEWER 2 for this?

Member

otizonaizit commented May 9, 2016

EDITOR
@heplesser: Can you be REVIEWER 1 for this?
@apdavison: Can you be REVIEWER 2 for this?

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison May 9, 2016

REVIEWER 2
@otizonaizit yes, I'd be happy to review this.

apdavison commented May 9, 2016

REVIEWER 2
@otizonaizit yes, I'd be happy to review this.

@FedericoV

This comment has been minimized.

Show comment
Hide comment
@FedericoV

FedericoV May 9, 2016

I might also be able to help if needed. It's Python and differential
equations.

On Mon, 9 May 2016 at 11:54 Tiziano Zito notifications@github.com wrote:

EDITOR
@heplesser https://github.com/heplesser: Can you be REVIEWER 1 for
this?
@apdavison https://github.com/apdavison: Can you be REVIEWER 2 for
this?


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#17 (comment)

I might also be able to help if needed. It's Python and differential
equations.

On Mon, 9 May 2016 at 11:54 Tiziano Zito notifications@github.com wrote:

EDITOR
@heplesser https://github.com/heplesser: Can you be REVIEWER 1 for
this?
@apdavison https://github.com/apdavison: Can you be REVIEWER 2 for
this?


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#17 (comment)

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit May 9, 2016

Member

Thanks Federico, I'd let @heplesser a bit of time to accept the review.
In case he is not available for this review I'll come back to your
offer.

On Mon 09 May, 04:11, Federico Vaggi notifications@github.com wrote:

I might also be able to help if needed. It's Python and differential
equations.

On Mon, 9 May 2016 at 11:54 Tiziano Zito notifications@github.com wrote:

EDITOR
@heplesser https://github.com/heplesser: Can you be REVIEWER 1 for
this?
@apdavison https://github.com/apdavison: Can you be REVIEWER 2 for
this?


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#17 (comment)


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#17 (comment)

Member

otizonaizit commented May 9, 2016

Thanks Federico, I'd let @heplesser a bit of time to accept the review.
In case he is not available for this review I'll come back to your
offer.

On Mon 09 May, 04:11, Federico Vaggi notifications@github.com wrote:

I might also be able to help if needed. It's Python and differential
equations.

On Mon, 9 May 2016 at 11:54 Tiziano Zito notifications@github.com wrote:

EDITOR
@heplesser https://github.com/heplesser: Can you be REVIEWER 1 for
this?
@apdavison https://github.com/apdavison: Can you be REVIEWER 2 for
this?


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#17 (comment)


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#17 (comment)

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser May 9, 2016

REVIEWER 1
@otizonaizit I'd be happy to review this.

heplesser commented May 9, 2016

REVIEWER 1
@otizonaizit I'd be happy to review this.

@rougier rougier changed the title from Review Request to Review Request: Detorakis May 10, 2016

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit May 10, 2016

Member

EDITOR
@gdetor: your paper is under review by @heplesser (REVIEWER 1) and @apdavison (REVIEWER 2). Stay tuned ;)

Member

otizonaizit commented May 10, 2016

EDITOR
@gdetor: your paper is under review by @heplesser (REVIEWER 1) and @apdavison (REVIEWER 2). Stay tuned ;)

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser May 12, 2016

REVIEWER 1

@gdetor I will begin with a few remarks and return with more detailed comments later.

First of all, your code ran out of the box and created the figures included in your manuscript.

While I think that you are quite close to re-implementing Wang's model (I have not yet compared the equations from the paper in detail to your code), I believe that more effort is needed to establish the quality of your model thoroughly. My understanding of the idea behind ReScience is that your implementation of Wang's model would become the gold standard for anyone who would want to use this model, including reimplementation in other software. Therefore, it is, in my opinion, essential that you verify as exactly as possible that your implementation reproduces Wang's results. Where discrepancies occur, they need to be analyzed and explained in detail.

Specifically, I think that it would much strengthen your model if you would recreate all figures and Table 1. To judge the quality of a model re-implementation, it is particularly interesting to see whether it responds to parameter changes in the same way as the original. Fig 1, bottom, Fig 2C, Table 1, and Fig 7, seem particularly relevant in that respect.

You write that you needed to adjust some parameter values slightly to match Wang's figures, and speculate that this may be due to your use of a different integration method. This is problematic. Clearly, it might be that some of the details of the responses shown in Wang's paper are as they are due to the specific numeric method used, and it might very well be that the method you use is better, in the sense that the solutions you obtain are closer to the mathematically correct solution. But it might also be due to other effects.

Therefore, I think it is important to first try to reproduce Wang as closely as possible, using the same integrator used originally. The precise step-size control is not given in the paper, but you could easily try the dopri5 4(5) order RK that comes with scipy.integrate.ode. This would allow you to explore to which degree your results depend on the integration method used. In general, I'd suggest that you allow the user to specify the integration method for easier testing. Where parameter adjustments are required, you should make explicit which parameters were changed from which value to which and with which effect.

heplesser commented May 12, 2016

REVIEWER 1

@gdetor I will begin with a few remarks and return with more detailed comments later.

First of all, your code ran out of the box and created the figures included in your manuscript.

While I think that you are quite close to re-implementing Wang's model (I have not yet compared the equations from the paper in detail to your code), I believe that more effort is needed to establish the quality of your model thoroughly. My understanding of the idea behind ReScience is that your implementation of Wang's model would become the gold standard for anyone who would want to use this model, including reimplementation in other software. Therefore, it is, in my opinion, essential that you verify as exactly as possible that your implementation reproduces Wang's results. Where discrepancies occur, they need to be analyzed and explained in detail.

Specifically, I think that it would much strengthen your model if you would recreate all figures and Table 1. To judge the quality of a model re-implementation, it is particularly interesting to see whether it responds to parameter changes in the same way as the original. Fig 1, bottom, Fig 2C, Table 1, and Fig 7, seem particularly relevant in that respect.

You write that you needed to adjust some parameter values slightly to match Wang's figures, and speculate that this may be due to your use of a different integration method. This is problematic. Clearly, it might be that some of the details of the responses shown in Wang's paper are as they are due to the specific numeric method used, and it might very well be that the method you use is better, in the sense that the solutions you obtain are closer to the mathematically correct solution. But it might also be due to other effects.

Therefore, I think it is important to first try to reproduce Wang as closely as possible, using the same integrator used originally. The precise step-size control is not given in the paper, but you could easily try the dopri5 4(5) order RK that comes with scipy.integrate.ode. This would allow you to explore to which degree your results depend on the integration method used. In general, I'd suggest that you allow the user to specify the integration method for easier testing. Where parameter adjustments are required, you should make explicit which parameters were changed from which value to which and with which effect.

@rougier rougier assigned otizonaizit and unassigned rougier May 14, 2016

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit May 18, 2016

Member

EDITOR
@apdavison: any chance you can review this by the end of the week? If not, let me know when you'll have time :)

Member

otizonaizit commented May 18, 2016

EDITOR
@apdavison: any chance you can review this by the end of the week? If not, let me know when you'll have time :)

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison May 18, 2016

REVIEWER 2
I've started the review (the code runs fine and creates the figures in the manuscript) but I'm at a conference tomorrow and Friday so can't finish it this week. I should have time on Monday (23rd).

apdavison commented May 18, 2016

REVIEWER 2
I've started the review (the code runs fine and creates the figures in the manuscript) but I'm at a conference tomorrow and Friday so can't finish it this week. I should have time on Monday (23rd).

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit May 18, 2016

Member

EDITOR
@apdavison: great, thank you!

Member

otizonaizit commented May 18, 2016

EDITOR
@apdavison: great, thank you!

code/neuron_model.py
+
+
+def loadParameters(fname):
+ """ Load all the necessary paremeters from a file.

This comment has been minimized.

code/neuron_model.py
+ """ Load all the necessary paremeters from a file.
+
+ | :param fname: File name containing all parameters
+ | :return: A dictionaty with all the parameters

This comment has been minimized.

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison May 23, 2016

REVIEWER 2
The code ran without first time without problems, and a visual inspection shows a clean, easy-to-read structure which is ideal for a reference implementation.

I am not, however, convinced that the results are quantitatively comparable to the original.

  1. The time axis in Figure 2 is 35 seconds in size. In the original Figure 3, however, the scale bar is 100 ms, which suggests the entire x-axis is about 1 second.
  2. It is difficult to compare the scales of Figure 3 and the original Figure 6, since the size of the axes is different. Figure 3 should be replotted on the same axes as the original.
  3. It is not sufficient to say that the integration method was different, and some of the parameters had to be changed to get the results to agree. If possible, the same integration method as the original, or one which is similar to it, should be used. If this would require extraordinary efforts, then it should at least be demonstrated that the results are consistent when (i) using two different integration methods and (ii) reducing the integration time step by a factor of ten.
  4. I do not think that reproduction of three figures showing membrane potential traces is sufficient to claim reproduction. Either Table 1 or Figure 7 or both should also be reproduced.

Three minor comments on the typesetting of the paper:

  1. there should always be a space between the numerical value and unit symbol (see http://physics.nist.gov/cuu/Units/checklist.html point 15) (I find the LaTeX "small space", \,, works well);
  2. unit symbols should be in roman type (http://physics.nist.gov/cuu/Units/checklist.html point 6) (e.g. \mathrm{mV})
  3. Subscripts should be in roman type if they are descriptive (e.g. I_{\mathrm{K}} since K is for potassium and is not a variable name). Italic type is for subscripts which represent variables (e.g. \beta_{n} is fine, since n is a variable of the model). (http://physics.nist.gov/cuu/Units/checklist.html point 7).

apdavison commented May 23, 2016

REVIEWER 2
The code ran without first time without problems, and a visual inspection shows a clean, easy-to-read structure which is ideal for a reference implementation.

I am not, however, convinced that the results are quantitatively comparable to the original.

  1. The time axis in Figure 2 is 35 seconds in size. In the original Figure 3, however, the scale bar is 100 ms, which suggests the entire x-axis is about 1 second.
  2. It is difficult to compare the scales of Figure 3 and the original Figure 6, since the size of the axes is different. Figure 3 should be replotted on the same axes as the original.
  3. It is not sufficient to say that the integration method was different, and some of the parameters had to be changed to get the results to agree. If possible, the same integration method as the original, or one which is similar to it, should be used. If this would require extraordinary efforts, then it should at least be demonstrated that the results are consistent when (i) using two different integration methods and (ii) reducing the integration time step by a factor of ten.
  4. I do not think that reproduction of three figures showing membrane potential traces is sufficient to claim reproduction. Either Table 1 or Figure 7 or both should also be reproduced.

Three minor comments on the typesetting of the paper:

  1. there should always be a space between the numerical value and unit symbol (see http://physics.nist.gov/cuu/Units/checklist.html point 15) (I find the LaTeX "small space", \,, works well);
  2. unit symbols should be in roman type (http://physics.nist.gov/cuu/Units/checklist.html point 6) (e.g. \mathrm{mV})
  3. Subscripts should be in roman type if they are descriptive (e.g. I_{\mathrm{K}} since K is for potassium and is not a variable name). Italic type is for subscripts which represent variables (e.g. \beta_{n} is fine, since n is a variable of the model). (http://physics.nist.gov/cuu/Units/checklist.html point 7).
@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit May 23, 2016

Member

EDITOR
@heplesser, @apdavison: thank you for your reviews!
@gdetor: Can you address the reviewers' comments?

Member

otizonaizit commented May 23, 2016

EDITOR
@heplesser, @apdavison: thank you for your reviews!
@gdetor: Can you address the reviewers' comments?

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor May 23, 2016

@apdavison @heplesser Thank you for the comments/suggestions.
@otizonaizit I already started addressing reviewers comments. Once everything's ready I will commit the changes. Thank you.

gdetor commented May 23, 2016

@apdavison @heplesser Thank you for the comments/suggestions.
@otizonaizit I already started addressing reviewers comments. Once everything's ready I will commit the changes. Thank you.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Jun 7, 2016

Member

@gdetor @otizonaizit Any progress ?

Member

rougier commented Jun 7, 2016

@gdetor @otizonaizit Any progress ?

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Jun 7, 2016

I'm still running some long-running simulations in order to validate the model. I estimate by the end of this week, I will submit the updated version.

gdetor commented Jun 7, 2016

I'm still running some long-running simulations in order to validate the model. I estimate by the end of this week, I will submit the updated version.

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Jun 14, 2016

Dear all,
Thank you again for your comments, corrections and suggestions. I submitted a revised version of the reference implementation following the comments of the reviewers.

  1. I recreated Figures 1, 2(partially), 3, 6, and 7.
  2. I changed all the figures time scales in order to be comparable to the original ones.
  3. Three integration method were tested (dopri5, Adams and BDF) and are now available as arguments in the code. All three methods give the same numerical results implying that is not the integration scheme responsible for any divergence from the original implementation. Furthermore, different time steps were tested without affecting the results.
  4. Any divergence from the original results has been addressed and discussed.
  5. The source code execution time is now quite long since it runs three parameters diagrams.

Best regards,
Georgios Detorakis

gdetor commented Jun 14, 2016

Dear all,
Thank you again for your comments, corrections and suggestions. I submitted a revised version of the reference implementation following the comments of the reviewers.

  1. I recreated Figures 1, 2(partially), 3, 6, and 7.
  2. I changed all the figures time scales in order to be comparable to the original ones.
  3. Three integration method were tested (dopri5, Adams and BDF) and are now available as arguments in the code. All three methods give the same numerical results implying that is not the integration scheme responsible for any divergence from the original implementation. Furthermore, different time steps were tested without affecting the results.
  4. Any divergence from the original results has been addressed and discussed.
  5. The source code execution time is now quite long since it runs three parameters diagrams.

Best regards,
Georgios Detorakis

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Jun 15, 2016

Member

EDITOR
@heplesser, @apdavison: could you have a second round of review? Thanks!

Member

otizonaizit commented Jun 15, 2016

EDITOR
@heplesser, @apdavison: could you have a second round of review? Thanks!

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Jun 22, 2016

Member

EDITOR
@heplesser, @apdavison: Any news? When do you think you could have a second round of review for the submission?

Member

otizonaizit commented Jun 22, 2016

EDITOR
@heplesser, @apdavison: Any news? When do you think you could have a second round of review for the submission?

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Jun 22, 2016

I hope to get around to it within a week.

On 22 Jun 2016, at 11:26, Tiziano Zito notifications@github.com wrote:

EDITOR
@heplesser, @apdavison: Any news? When do you think you could have a second round of review for the submission?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Dr. Hans Ekkehard Plesser, Associate Professor
Section Head

Dept. of Mathematical Sciences and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560
Email hans.ekkehard.plesser@nmbu.no
Home http://arken.umb.no/~plesser

I hope to get around to it within a week.

On 22 Jun 2016, at 11:26, Tiziano Zito notifications@github.com wrote:

EDITOR
@heplesser, @apdavison: Any news? When do you think you could have a second round of review for the submission?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Dr. Hans Ekkehard Plesser, Associate Professor
Section Head

Dept. of Mathematical Sciences and Technology
Norwegian University of Life Sciences
PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560
Email hans.ekkehard.plesser@nmbu.no
Home http://arken.umb.no/~plesser

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison Jun 24, 2016

Well, the simulations are still running... :-)

Well, the simulations are still running... :-)

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Jul 4, 2016

Member

EDITOR-IN-CHIEF

@otizonaizit Any update ?

Member

rougier commented Jul 4, 2016

EDITOR-IN-CHIEF

@otizonaizit Any update ?

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Jul 4, 2016

REVIEWER 1
@gdetor

  • Table 2, top row: Do you mean "2.5 x period" or "2 x period, 5 x period"?
  • In footnote 1 on p 2, you write that "we decided to use BDF". But if simulations.py, you use Adams for all figures except for Fig 5, where you use BDF.
  • You write on p 2 "BDF and Adams provide exactly the same numerical results as [dopri5]". This is not true. I ran the code for Fig 2 using Adams and BDF with vode, dopri5, and lsoda (for Iapp=-0.6). All generated different results around spikes. If one, e.g., set a spike threshold at V = 0 mV, these differences will occasionally lead to shifts in spike time by one time step.
  • The time axis in your Fig 2 is from 0 to 2 s. This raises two questions: When running the simulation code for Fig 2 and plotting the data directly, there is a strong transitory effect during the first 25 ms. This is completely absent in your Fig 2. It is plausible that Wang left out the transition in his figure, and it is justifiable in your work, but you need to be explicit on this.
  • Time in your Fig 2 runs from 0 to 2 s, while in the corresponding Fig 3 in the original paper, the horizontal axis spans 1 s (estimate). Thus, your figure seems off by a factor 2. How can you claim that "there as no difference between the reference implementation and the original one"?
  • In Fig 2, you should also include all cases that Wang had in his figure, and draw all lines in the same color---no need for color maps in graphs containing a single line. The I_app labels could be placed just to the right of the plot, not in it, and would look better without frames.
  • Fig 1: This paper is about reproducing Wang's results. This would be much easier to check if you chose figures that were as close as at all possible to Wang's figures. So please use the same axes labelling as Wang, and use a contour line graph instead of a greyscale graph. Your greyscale graph shows a fuzzy upper edge of the main triangle structure, with "tongues" sticking out to the right. This seems to be in contrast to Wang's figure. Is this an artifact in your simulations? Did Wang possibly coarse-grain the original figure? Unfortunately, the original paper does not contain explicit information on how many P0 and P1 values were evaluated to create that figure.
  • Why do you still not reproduce Wang's Figs 2, 4 and 5?
  • Fig 3 (Wang's Fig 6): Please use the same axis limits and proportions as Wang for comparability. Comparing the voltage traces visually, it seems to me that your figure shows noticeably fewer spikes, while the h-signal is significantly more regular than in Wang's figure.
  • Fig 4 (Wang's Fig 7): If I understand your figure caption correctly, then the top left figure corresponds to Wang's 7A, while the top right corresponds to Wang's 7B. You describe the curves as "qualitatively similar to the original ones". Again, comparison is difficult because you use different axis limits and do not show the individual data points as in the original. But it appears to me that the reproduction of 7B in particular is quite different from Wang's figure: You figure shows a very clear stepping pattern, while Wang's figure shows a smooth curve. This smoothness of Wang's curves may be due to a limited number of data points.

heplesser commented Jul 4, 2016

REVIEWER 1
@gdetor

  • Table 2, top row: Do you mean "2.5 x period" or "2 x period, 5 x period"?
  • In footnote 1 on p 2, you write that "we decided to use BDF". But if simulations.py, you use Adams for all figures except for Fig 5, where you use BDF.
  • You write on p 2 "BDF and Adams provide exactly the same numerical results as [dopri5]". This is not true. I ran the code for Fig 2 using Adams and BDF with vode, dopri5, and lsoda (for Iapp=-0.6). All generated different results around spikes. If one, e.g., set a spike threshold at V = 0 mV, these differences will occasionally lead to shifts in spike time by one time step.
  • The time axis in your Fig 2 is from 0 to 2 s. This raises two questions: When running the simulation code for Fig 2 and plotting the data directly, there is a strong transitory effect during the first 25 ms. This is completely absent in your Fig 2. It is plausible that Wang left out the transition in his figure, and it is justifiable in your work, but you need to be explicit on this.
  • Time in your Fig 2 runs from 0 to 2 s, while in the corresponding Fig 3 in the original paper, the horizontal axis spans 1 s (estimate). Thus, your figure seems off by a factor 2. How can you claim that "there as no difference between the reference implementation and the original one"?
  • In Fig 2, you should also include all cases that Wang had in his figure, and draw all lines in the same color---no need for color maps in graphs containing a single line. The I_app labels could be placed just to the right of the plot, not in it, and would look better without frames.
  • Fig 1: This paper is about reproducing Wang's results. This would be much easier to check if you chose figures that were as close as at all possible to Wang's figures. So please use the same axes labelling as Wang, and use a contour line graph instead of a greyscale graph. Your greyscale graph shows a fuzzy upper edge of the main triangle structure, with "tongues" sticking out to the right. This seems to be in contrast to Wang's figure. Is this an artifact in your simulations? Did Wang possibly coarse-grain the original figure? Unfortunately, the original paper does not contain explicit information on how many P0 and P1 values were evaluated to create that figure.
  • Why do you still not reproduce Wang's Figs 2, 4 and 5?
  • Fig 3 (Wang's Fig 6): Please use the same axis limits and proportions as Wang for comparability. Comparing the voltage traces visually, it seems to me that your figure shows noticeably fewer spikes, while the h-signal is significantly more regular than in Wang's figure.
  • Fig 4 (Wang's Fig 7): If I understand your figure caption correctly, then the top left figure corresponds to Wang's 7A, while the top right corresponds to Wang's 7B. You describe the curves as "qualitatively similar to the original ones". Again, comparison is difficult because you use different axis limits and do not show the individual data points as in the original. But it appears to me that the reproduction of 7B in particular is quite different from Wang's figure: You figure shows a very clear stepping pattern, while Wang's figure shows a smooth curve. This smoothness of Wang's curves may be due to a limited number of data points.
article/detorakis-2016.tex
+ significant differences for the randomly picked up spike waveform shown in
+ this figure (in general we detected differences up to $5\, \Rm{mV}$, data
+ not shown here). Continuous line -- \emph{dopri5}, dashed line -- \emph{Adams},
+ and dashed-dotted line -- \emph{BDF}.}

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

Move line-style info up. Note that -- gives a from-to dash ("5--7 pages"), while --- gives the em-dash you probably wanted here; according to Knuth, there should be no space around em-dashes.

@heplesser

heplesser Aug 9, 2016

Move line-style info up. Note that -- gives a from-to dash ("5--7 pages"), while --- gives the em-dash you probably wanted here; according to Knuth, there should be no space around em-dashes.

article/detorakis-2016.tex
+implementation to rhythmic hyperpolarization. In \cite{wang:1994} this is
+illustrated in Fig~$1$\footnote{From now and then all the
+ figures of the original article will be referred as Fig} of the original
+article.

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

This is double-up: "In Wang (1994) ... of the original article". Wang is the original article.

@heplesser

heplesser Aug 9, 2016

This is double-up: "In Wang (1994) ... of the original article". Wang is the original article.

article/detorakis-2016.tex
+is omitted in the rest of the figures in this work). It is apparent that right
+and left panels are in an agreement indicating that the reference model
+produces reliable results.
+The author in \cite{wang:1994} shows in Fig~$1$ a shaded are where the number

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

Drop "The author in", just "Wang (1994) shows

@heplesser

heplesser Aug 9, 2016

Drop "The author in", just "Wang (1994) shows

article/detorakis-2016.tex
+produces reliable results.
+The author in \cite{wang:1994} shows in Fig~$1$ a shaded are where the number
+of spikes is $0.5$. This number appears if one computes the average of
+suprathreshold and subthreshold spikes in one period.

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

"This number appears ...": what do you mean by "appears"?

@heplesser

heplesser Aug 9, 2016

"This number appears ...": what do you mean by "appears"?

code/neuron_model.py
+ | :param inp_type: Input signal type ['const' - constant current
+ 'periodic' - periodic pulses]
+ 'pulse' - one pulse at the
+ beggining of simulation

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

"beginning"

article/detorakis-2016.tex
+Three different methods have been tested in this work (\emph{dopri5},
+\emph{Adams}, \emph{BDF}, \cite{ascher:1998}). \emph{dopri5} is the closest
+numerical method to the one used by the author in the original article (the
+author has numerically integrated the system of the four ODEs by using a

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

"(the author has ...": it is not entirely clear whether "the author" here refers to Wang or to you. Write "Wang" for clarity" or cut it to just "(fifth-order adaptive stepsize Runge-Kutta)".

@heplesser

heplesser Aug 9, 2016

"(the author has ...": it is not entirely clear whether "the author" here refers to Wang or to you. Write "Wang" for clarity" or cut it to just "(fifth-order adaptive stepsize Runge-Kutta)".

article/detorakis-2016.tex
+ three methods. The membrane potential amplitude was sometimes different
+ for the same spike events, however that difference was not too high as
+ Figure~\ref{Fig:1} shows (up to $4\, \Rm{mV}$ at maximum). Thus we decided
+ to use the \emph{Adams} method.}.

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

I think this is important information that should not be hidden in a footnote.

@heplesser

heplesser Aug 9, 2016

I think this is important information that should not be hidden in a footnote.

article/detorakis-2016.tex
+ (see text for more details). The amplitude for the three methods have no
+ significant differences for the randomly picked up spike waveform shown in
+ this figure (in general we detected differences up to $5\, \Rm{mV}$, data
+ not shown here). Continuous line -- \emph{dopri5}, dashed line -- \emph{Adams},

This comment has been minimized.

@heplesser

heplesser Aug 9, 2016

I am confused: You show a graph in which the solution of all three methods follow each other closely, but then you mention in passing in the legend that you observed differences of up to 5 mV, which is about 10% of the entire range of voltage excursions. Shouldn't you show such a worst-case example as well?

@heplesser

heplesser Aug 9, 2016

I am confused: You show a graph in which the solution of all three methods follow each other closely, but then you mention in passing in the legend that you observed differences of up to 5 mV, which is about 10% of the entire range of voltage excursions. Shouldn't you show such a worst-case example as well?

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Aug 9, 2016

@gdetor Thank you for your significant improvements to manuscript and code. I still think that some further improvements are necessary as noted in inline comments and below.

I would also like to ask the editors (@otizonaizit, @rougier) for principal input on how to define successful reproduction in view of limited information in and about the original work. I will detail my concerns below.

Wang Fig 1/your Fig 2:

  • Your figure has different axis ticks than the original, and the individual traces are different; especially, the time axis is too long for the middle case, suppressing detail, and too short for the bottom case, leaving open the question of whether a second burst will follow. Figures would be much more comparable if you would attempt to imitate the original figure layout as closely as possible.
  • Why is the shaded area (0.5 spikes) missing in your figure?
  • Your figure looks quite different from Wang's. E.g.,
    • there is no "tongue" stretching from the "4 spikes" area towards the origin
    • the right edge of the 3 and 4 spikes areas is much closer to the y-axis and much steeper than in the original
    • there is a "thumb" sticking out of the 1 spike area towards the upper right
    • your curves are stepped, Wang's are smooth.
      That said, I find Wang's curves rather too smooth, taking into account the available computer power in the early 1990s. Unfortunately, Wang does not provide any detail on the grid resolution used to generate the original figure, and whether results were averaged across different initial conditions or so, which might have smoothed curves. But the discrepancy needs to be discussed, and the question of what constitutes successful reproduction addressed (i.e., why, in your scientific judgement, the figure constitutes reproduction in spite of the difference in detail).

Your Fig 4:
Why don't you place the sub-figures to the right in the same order as in the original figure?

Your Fig 5:
Could you choose the same aspect ratio as in the original figure to make figures even more comparable?

Your Fig 6:

  • Please add A and B labels
  • In your right panel, the blue curve is flat across three data points (I_app -0.9, -1.0, -1.2 approx), while in the original figure, the value around I_app = -1.2 seems to be clearly higher than for -0.1. This seems to be an explicit difference between the original result and your results. Where does it come from?
  • You need to discuss also here why you think the fact that your curves are stepped, while Wang's are smooth, do not constitute a problem for reproducibility. Again, Wang's curves may look too smooth, and one should probably stick to just those data points explicitly marked in his Fig 7. Could you plot those in your figure for direct comparison?

@gdetor Thank you for your significant improvements to manuscript and code. I still think that some further improvements are necessary as noted in inline comments and below.

I would also like to ask the editors (@otizonaizit, @rougier) for principal input on how to define successful reproduction in view of limited information in and about the original work. I will detail my concerns below.

Wang Fig 1/your Fig 2:

  • Your figure has different axis ticks than the original, and the individual traces are different; especially, the time axis is too long for the middle case, suppressing detail, and too short for the bottom case, leaving open the question of whether a second burst will follow. Figures would be much more comparable if you would attempt to imitate the original figure layout as closely as possible.
  • Why is the shaded area (0.5 spikes) missing in your figure?
  • Your figure looks quite different from Wang's. E.g.,
    • there is no "tongue" stretching from the "4 spikes" area towards the origin
    • the right edge of the 3 and 4 spikes areas is much closer to the y-axis and much steeper than in the original
    • there is a "thumb" sticking out of the 1 spike area towards the upper right
    • your curves are stepped, Wang's are smooth.
      That said, I find Wang's curves rather too smooth, taking into account the available computer power in the early 1990s. Unfortunately, Wang does not provide any detail on the grid resolution used to generate the original figure, and whether results were averaged across different initial conditions or so, which might have smoothed curves. But the discrepancy needs to be discussed, and the question of what constitutes successful reproduction addressed (i.e., why, in your scientific judgement, the figure constitutes reproduction in spite of the difference in detail).

Your Fig 4:
Why don't you place the sub-figures to the right in the same order as in the original figure?

Your Fig 5:
Could you choose the same aspect ratio as in the original figure to make figures even more comparable?

Your Fig 6:

  • Please add A and B labels
  • In your right panel, the blue curve is flat across three data points (I_app -0.9, -1.0, -1.2 approx), while in the original figure, the value around I_app = -1.2 seems to be clearly higher than for -0.1. This seems to be an explicit difference between the original result and your results. Where does it come from?
  • You need to discuss also here why you think the fact that your curves are stepped, while Wang's are smooth, do not constitute a problem for reproducibility. Again, Wang's curves may look too smooth, and one should probably stick to just those data points explicitly marked in his Fig 7. Could you plot those in your figure for direct comparison?
@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Aug 10, 2016

Member

EDITOR-IN-CHIEF

@heplesser @otizonaizit There is no definitive criterion for assessing a replication is successful. In some case it is purely qualitative while in some others, we would need the exact replication of numerical results. In this specific case, it is mostly the decision of the author, the reviewer and the editor to decide if the original results can be reproduced. Furthermore, I suspect there might exist some numerical discrepancies that cannot be solved without full access to the original source code.
Look for instance at http://arxiv.org/abs/1605.04339 and you will realize the difficulty in having a bit for bit reproduction.

And of course, we can also open an issue after this review to see if we can define some a set of formal criterion to decide if a replication is ok or not.

Member

rougier commented Aug 10, 2016

EDITOR-IN-CHIEF

@heplesser @otizonaizit There is no definitive criterion for assessing a replication is successful. In some case it is purely qualitative while in some others, we would need the exact replication of numerical results. In this specific case, it is mostly the decision of the author, the reviewer and the editor to decide if the original results can be reproduced. Furthermore, I suspect there might exist some numerical discrepancies that cannot be solved without full access to the original source code.
Look for instance at http://arxiv.org/abs/1605.04339 and you will realize the difficulty in having a bit for bit reproduction.

And of course, we can also open an issue after this review to see if we can define some a set of formal criterion to decide if a replication is ok or not.

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Aug 11, 2016

Member

EDITOR

@heplesser, @rougier: I think that the manuscript can be considered to successfully replicate the original paper, provided the authors address @heplesser comments either implementing his suggestions or explaining why they can't be implemented. I am still waiting for @apdavison final recommendation to be able to come to a decision, but given the tone of his review I think we are very close to a positive outcome, @gdetor.

Member

otizonaizit commented Aug 11, 2016

EDITOR

@heplesser, @rougier: I think that the manuscript can be considered to successfully replicate the original paper, provided the authors address @heplesser comments either implementing his suggestions or explaining why they can't be implemented. I am still waiting for @apdavison final recommendation to be able to come to a decision, but given the tone of his review I think we are very close to a positive outcome, @gdetor.

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison Aug 17, 2016

I recommend acceptance. Concerning Figure 2, I doubt it is possible to more closely reproduce the original Fig 1 in the absence of information about which method was used to produce the smooth curves, although I agree with @heplesser that some comment should be made about this in the manuscript.

I recommend acceptance. Concerning Figure 2, I doubt it is possible to more closely reproduce the original Fig 1 in the absence of information about which method was used to produce the smooth curves, although I agree with @heplesser that some comment should be made about this in the manuscript.

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Aug 18, 2016

Dear all,
Thank you again for your comments/suggestions/corrections. All the comments have been addressed (code, text, and figures). I think that now almost all of the figures are quite close to the original ones except my second one (first figure in Wang). I tried a high resolution (100 x 100) discretization of the parameter space, and now the curves are smoother than previous ones but still there are some differences.

Best regards,
Georgios Detorakis

gdetor commented Aug 18, 2016

Dear all,
Thank you again for your comments/suggestions/corrections. All the comments have been addressed (code, text, and figures). I think that now almost all of the figures are quite close to the original ones except my second one (first figure in Wang). I tried a high resolution (100 x 100) discretization of the parameter space, and now the curves are smoother than previous ones but still there are some differences.

Best regards,
Georgios Detorakis

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Aug 19, 2016

Member

EDITOR
@gdetor: congratulations, I hereby accept the submission
@heplesser, @apdavison: thanks to the reviewers for their helpful comments, I think they helped in improving the paper significantly.

@gdetor, @rougier: I am currently mostly offline or with very flaky internet connection. I'll publish the paper in two weeks when I'm back to civilization if this is fine with you.

Member

otizonaizit commented Aug 19, 2016

EDITOR
@gdetor: congratulations, I hereby accept the submission
@heplesser, @apdavison: thanks to the reviewers for their helpful comments, I think they helped in improving the paper significantly.

@gdetor, @rougier: I am currently mostly offline or with very flaky internet connection. I'll publish the paper in two weeks when I'm back to civilization if this is fine with you.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Aug 20, 2016

Member

@otizonaizit Fine for me. The publication process is still not straightforward so you might need my help but it would be good to have your comments on what could be modified.

Member

rougier commented Aug 20, 2016

@otizonaizit Fine for me. The publication process is still not straightforward so you might need my help but it would be good to have your comments on what could be modified.

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Aug 22, 2016

Member

Hi @gdetor, Reviewer 1 noticed that I did not wait for his recommendation before deciding, and I think his further comments to the manuscript are definitely worth addressing. @heplesser: can you please post your further comments? @gdetor: could you please address @heplesser comments so that we can finally accept and publish the paper? Thanks!

Member

otizonaizit commented Aug 22, 2016

Hi @gdetor, Reviewer 1 noticed that I did not wait for his recommendation before deciding, and I think his further comments to the manuscript are definitely worth addressing. @heplesser: can you please post your further comments? @gdetor: could you please address @heplesser comments so that we can finally accept and publish the paper? Thanks!

@rougier rougier removed the 03 - Accepted label Aug 22, 2016

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Aug 22, 2016

@otizonaizit Sure, I'll wait for the comments.

gdetor commented Aug 22, 2016

@otizonaizit Sure, I'll wait for the comments.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Aug 22, 2016

@gdetor I believe that the paper is essentially sound and should be published, but there are a few things to fix:

In the new Figure 1, the horizontal axis does not show "Error", but "Voltage Difference" between the solutions. The unit of measurement should be given, which, I believe, is mV here. On the vertical axis, the unit should not be Hz, which applies to periodic processes only. If I read the code correctly, data is taken by binning values from simulation over 6000 ms into 30 bins, so what is shown is number of occurances of voltage differences over 6 seconds. I am also not sure that this diagram really provides useful information. Maybe it would be better to plot one solution as function of the other solution. In case of perfect agreement, one would just get a straight line along the diagonal, deviations show where/how the different solvers behave differently.

I am also not sure what you mean by "overlapping spike events".

Figure 6 now looks much better than in the previous version, in fine agreement with the original. From the changes in code/params/params_figure6a.cfg

-phi_K = 28.6
+phi_K = 28.5714285714

it appears that this very delicate adjustment in phi_K was required to reach agreement (phi_K is phi_n in the paper). The latter value is indeed the correct decimal value for 200/7 from the paper (it would be useful to add this as a comment in the config file). But I think it would be very useful for readers and people who work with the code to know that a change in phi_K by about 0.1% will have distinct effects on the behavior of the neuron.

Furthermore, Table 5 of the manuscript gives a value of $\phi_n = 28.5$. That value is according to the code/params/*.cfg files used for figures 1, 3 and 5, while figure 2 uses 28.6, and figure 4, 6 and 7 use 200/7. Given the significant effect of minuscule changes, this is not acceptable. All figures should be based on 200/7, and that value should be given in the Table (Btw, does your config-file format allow for comments? It is not immediately obvious that 200/7 == 28.5714285714)

This raises another concern: Are all other parameters correct as given in the Tables?

Finally, a figure reference is incorrect (Figure ??) in the conclusions, and the English in the parts of the text that came in with the most recent revision would benefit from some polishing.

@gdetor I believe that the paper is essentially sound and should be published, but there are a few things to fix:

In the new Figure 1, the horizontal axis does not show "Error", but "Voltage Difference" between the solutions. The unit of measurement should be given, which, I believe, is mV here. On the vertical axis, the unit should not be Hz, which applies to periodic processes only. If I read the code correctly, data is taken by binning values from simulation over 6000 ms into 30 bins, so what is shown is number of occurances of voltage differences over 6 seconds. I am also not sure that this diagram really provides useful information. Maybe it would be better to plot one solution as function of the other solution. In case of perfect agreement, one would just get a straight line along the diagonal, deviations show where/how the different solvers behave differently.

I am also not sure what you mean by "overlapping spike events".

Figure 6 now looks much better than in the previous version, in fine agreement with the original. From the changes in code/params/params_figure6a.cfg

-phi_K = 28.6
+phi_K = 28.5714285714

it appears that this very delicate adjustment in phi_K was required to reach agreement (phi_K is phi_n in the paper). The latter value is indeed the correct decimal value for 200/7 from the paper (it would be useful to add this as a comment in the config file). But I think it would be very useful for readers and people who work with the code to know that a change in phi_K by about 0.1% will have distinct effects on the behavior of the neuron.

Furthermore, Table 5 of the manuscript gives a value of $\phi_n = 28.5$. That value is according to the code/params/*.cfg files used for figures 1, 3 and 5, while figure 2 uses 28.6, and figure 4, 6 and 7 use 200/7. Given the significant effect of minuscule changes, this is not acceptable. All figures should be based on 200/7, and that value should be given in the Table (Btw, does your config-file format allow for comments? It is not immediately obvious that 200/7 == 28.5714285714)

This raises another concern: Are all other parameters correct as given in the Tables?

Finally, a figure reference is incorrect (Figure ??) in the conclusions, and the English in the parts of the text that came in with the most recent revision would benefit from some polishing.

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Aug 25, 2016

Dear all,
I've just committed new corrections according to reviewers comments.

Furthermore,

  1. In the new Figure 1, the horizontal axis does not show "Error", but "Voltage Difference" between the solutions. The unit of measurement should be given, which, I believe, is mV here. On the vertical axis, the unit should not be Hz, which applies to periodic processes only. If I read the code correctly, data is taken by binning values from simulation over 6000 ms into 30 bins, so what is shown is number of occurances of voltage differences over 6 seconds. I am also not sure that this diagram really provides useful information. Maybe it would be better to plot one solution as function of the other solution. In case of perfect agreement, one would just get a straight line along the diagonal, deviations show where/how the different solvers behave differently.

Figure 1 has now changed according to reviewer's suggestion.

  1. Figure 6 now looks much better than in the previous version, in fine agreement with the original. From the changes in code/params/params_figure6a.cfg

-phi_K = 28.6
+phi_K = 28.5714285714
it appears that this very delicate adjustment in phi_K was required to reach agreement (phi_K is phi_n in the paper). The latter value is indeed the correct decimal value for 200/7 from the paper (it would be useful to add this as a comment in the config file). But I think it would be very useful for readers and people who work with the code to know that a change in phi_K by about 0.1% will have distinct effects on the behavior of the neuron.

The problem was not the value of phi (since I tried several simulations wth both values 28.6 and 28.5714285714) but the total simulation time. The signal was too short for any further analysis. By increasing simulation time everything was better.

  1. Furthermore, Table 5 of the manuscript gives a value of $\phi_n = 28.5$. That value is according to the code/params/*.cfg files used for figures 1, 3 and 5, while figure 2 uses 28.6, and figure 4, 6 and 7 use 200/7. Given the significant effect of minuscule changes, this is not acceptable. All figures should be based on 200/7, and that value should be given in the Table (Btw, does your config-file format allow for comments? It is not immediately obvious that 200/7 == 28.5714285714)

Thank you for these comments. I found out that due to previous corrections and modifications some of the values in Table 5 were wrong. I double checked and corrected all the wrong values.

gdetor commented Aug 25, 2016

Dear all,
I've just committed new corrections according to reviewers comments.

Furthermore,

  1. In the new Figure 1, the horizontal axis does not show "Error", but "Voltage Difference" between the solutions. The unit of measurement should be given, which, I believe, is mV here. On the vertical axis, the unit should not be Hz, which applies to periodic processes only. If I read the code correctly, data is taken by binning values from simulation over 6000 ms into 30 bins, so what is shown is number of occurances of voltage differences over 6 seconds. I am also not sure that this diagram really provides useful information. Maybe it would be better to plot one solution as function of the other solution. In case of perfect agreement, one would just get a straight line along the diagonal, deviations show where/how the different solvers behave differently.

Figure 1 has now changed according to reviewer's suggestion.

  1. Figure 6 now looks much better than in the previous version, in fine agreement with the original. From the changes in code/params/params_figure6a.cfg

-phi_K = 28.6
+phi_K = 28.5714285714
it appears that this very delicate adjustment in phi_K was required to reach agreement (phi_K is phi_n in the paper). The latter value is indeed the correct decimal value for 200/7 from the paper (it would be useful to add this as a comment in the config file). But I think it would be very useful for readers and people who work with the code to know that a change in phi_K by about 0.1% will have distinct effects on the behavior of the neuron.

The problem was not the value of phi (since I tried several simulations wth both values 28.6 and 28.5714285714) but the total simulation time. The signal was too short for any further analysis. By increasing simulation time everything was better.

  1. Furthermore, Table 5 of the manuscript gives a value of $\phi_n = 28.5$. That value is according to the code/params/*.cfg files used for figures 1, 3 and 5, while figure 2 uses 28.6, and figure 4, 6 and 7 use 200/7. Given the significant effect of minuscule changes, this is not acceptable. All figures should be based on 200/7, and that value should be given in the Table (Btw, does your config-file format allow for comments? It is not immediately obvious that 200/7 == 28.5714285714)

Thank you for these comments. I found out that due to previous corrections and modifications some of the values in Table 5 were wrong. I double checked and corrected all the wrong values.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Aug 26, 2016

@gdetor Thank you very much for your revisions! I must admit it is a relieve to see that the difference in Fig 6 was due to simulation time, not due to tiny changes in phi_K. I think it would be useful to point out in the discussion of that figure (and of figure 2) that one needs to collect enough data to get results close to the originals. This is a valuable insight from your reproduction efforts and should not be hidden just in a table.

Concerning Table 2, I would us "Simulated times", not "Simulation times". The latter could be misunderstood as the time it took to run the simulation. For Fig 2, it should be $15\times\text{period}$, to that "period" is set as roman text.

Concerning Fig 1: It makes things look much worse than they are because you are connecting the dots. If you don't, it looks much better (see attached notebook); there, I shade an area +-2mV from the diagonal in addition.

But note that you code for checking correctness of spikes is not correct:

(dopri[spks1,0]-adams[spks2,0]).sum())

The sum always ends up zero, even though there are differences in spike times. For Adams vs DoPri5, only a single spike is one time step (0.05 ms) late, while for BDF vs DoPri5 more than half the spikes are 0.05ms late. But the conclusion seems sensible that Adams gives comparable spike trains and can be used.

detorakis_test.ipynb.zip

@gdetor Thank you very much for your revisions! I must admit it is a relieve to see that the difference in Fig 6 was due to simulation time, not due to tiny changes in phi_K. I think it would be useful to point out in the discussion of that figure (and of figure 2) that one needs to collect enough data to get results close to the originals. This is a valuable insight from your reproduction efforts and should not be hidden just in a table.

Concerning Table 2, I would us "Simulated times", not "Simulation times". The latter could be misunderstood as the time it took to run the simulation. For Fig 2, it should be $15\times\text{period}$, to that "period" is set as roman text.

Concerning Fig 1: It makes things look much worse than they are because you are connecting the dots. If you don't, it looks much better (see attached notebook); there, I shade an area +-2mV from the diagonal in addition.

But note that you code for checking correctness of spikes is not correct:

(dopri[spks1,0]-adams[spks2,0]).sum())

The sum always ends up zero, even though there are differences in spike times. For Adams vs DoPri5, only a single spike is one time step (0.05 ms) late, while for BDF vs DoPri5 more than half the spikes are 0.05ms late. But the conclusion seems sensible that Adams gives comparable spike trains and can be used.

detorakis_test.ipynb.zip

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Aug 26, 2016

Thank you again for those comments. I already addressed them and I committed the new files.

gdetor commented Aug 26, 2016

Thank you again for those comments. I already addressed them and I committed the new files.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Aug 28, 2016

@otizonaizit @gdetor Thank you for the revision. The paper now successfully reproduces the original results and I recommend acceptance. I just suggest two small language fixes. On p 2., "The most stroking difference found for the amplitude of membrane potential" should be "The most striking difference is found ...", and in Table 2, also the table caption should be changed to "Simulated time".

@otizonaizit @gdetor Thank you for the revision. The paper now successfully reproduces the original results and I recommend acceptance. I just suggest two small language fixes. On p 2., "The most stroking difference found for the amplitude of membrane potential" should be "The most striking difference is found ...", and in Table 2, also the table caption should be changed to "Simulated time".

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Aug 29, 2016

Dear all,
Thank you once again. I corrected the two typos and the final version is now in the repository.

gdetor commented Aug 29, 2016

Dear all,
Thank you once again. I corrected the two typos and the final version is now in the repository.

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Aug 29, 2016

Member

EDITOR
@gdetor: congratulations, I hereby accept the submission, this time for real ;-)
@heplesser, @apdavison: thanks to the reviewers again for their helpful comments.

@gdetor, @rougier: I am currently mostly offline or with very flaky internet connection. I'll publish the paper in next week when I'm back to civilization if this is fine with you.

Member

otizonaizit commented Aug 29, 2016

EDITOR
@gdetor: congratulations, I hereby accept the submission, this time for real ;-)
@heplesser, @apdavison: thanks to the reviewers again for their helpful comments.

@gdetor, @rougier: I am currently mostly offline or with very flaky internet connection. I'll publish the paper in next week when I'm back to civilization if this is fine with you.

@ReScience ReScience locked and limited conversation to collaborators Sep 7, 2016

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Sep 7, 2016

Member

@gdetor: can you give us some keywords to use for the publication? Right now I have: Neuroscience, Python, Replication. Some more detailed keyword would help. Thanks!

Member

otizonaizit commented Sep 7, 2016

@gdetor: can you give us some keywords to use for the publication? Right now I have: Neuroscience, Python, Replication. Some more detailed keyword would help. Thanks!

@otizonaizit

This comment has been minimized.

Show comment
Hide comment
@otizonaizit

otizonaizit Sep 7, 2016

Member

This submission has been accepted for publication, and has been published and appeared at
http://rescience.github.io/read/

DOI

Member

otizonaizit commented Sep 7, 2016

This submission has been accepted for publication, and has been published and appeared at
http://rescience.github.io/read/

DOI

@otizonaizit otizonaizit closed this Sep 7, 2016

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Sep 7, 2016

@otizonaizit Some extra keywords: Conductance-based model, Thalamic relay neurons, Intermittent phase-locking, Spindle oscillation, Delta oscillation

gdetor commented Sep 7, 2016

@otizonaizit Some extra keywords: Conductance-based model, Thalamic relay neurons, Intermittent phase-locking, Spindle oscillation, Delta oscillation

@damiendr

This comment has been minimized.

Show comment
Hide comment
@damiendr

damiendr Sep 7, 2016

Just noticed that the published PDF does not include the last commit for the typos.

damiendr commented Sep 7, 2016

Just noticed that the published PDF does not include the last commit for the typos.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Sep 7, 2016

Member

@damiendr Thanks !
@otizonaizit Do you where we messed up things ?

Member

rougier commented Sep 7, 2016

@damiendr Thanks !
@otizonaizit Do you where we messed up things ?

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Sep 8, 2016

Member

@damiendr Should be fixed by now.

Member

rougier commented Sep 8, 2016

@damiendr Should be fixed by now.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.