Review Request : M. Topalidou, N. Rougier

[rougier]

AUTHOR

Dear @ReScience/editors,

I request a review for the reproduction of the following paper:

• Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study, M. Guthrie, A. Leblois, A. Garenne, and T. Boraud, Journal of Neurophysiology, 109, 2013.

I believed the original results have been faithfully reproduced as explained in the accompanying article.

The repository lives at https://github.com/rougier/ReScience-submission/tree/topalidou-rougier

$ git clone https://github.com/rougier/ReScience-submission.git
$ cd ReScience-submission
$ git checkout topalidou-rougier
$ cd code

---

EDITOR

• Editor acknowledgment
• Review 1 started (@MehdiKhamassi)
• Review 2 started (@benoit-girard)
• Review 1 decision [accept]
• Review 2 decision [accept]
• Editor decision [accept]

[otizonaizit]

EDITOR

@rougier: I am going to edit this submission, thanks!
@benoit-girard, @MehdiKhamassi: Would you be able to review this submission? For the reviewing criteria you can have a look at the Reviewer Guidelines. Please accept or reject this review by commenting on this pull request. If you need assistance with the review process on GitHub, don't hesitate to contact me :)

[MehdiKhamassi]

REVIEWER 1

Review accepted

[benoit-girard]

REVIEWER 2

Review accepted. I am not familiar with GitHub, so I willprobably have a lot of questions :)

[benoit-girard]

REVIEWER 2

when I run the single-trial.py, the cortical activity does not go back to rest levels of activity after trial stop (after 2.5s), contrarily to what is reported in fig1.
figure-1.pdf

The 250-trial test gives results similar to what is presented in the paper, however the program ends abnormally :

[...]
Experiment 250: 0.783
Traceback (most recent call last):
File "250-simulations.py", line 254, in
sys.exit()
NameError: name 'sys' is not defined

sys has not been imported where it should have been?

[benoit-girard]

REVIEWER 2
I checked (superficially) that the provided code does not appear to simply write a predetermined set of values in log files, but does indeed implement the model. How deep should this check be? @otizonaizit

[rougier]

AUTHOR

In the single trial, I think we may have forgot to actually stop the trial. This will be corrected.

In the 250 trials test, we forgot to import the sys namespace. This will be corrected.

[rougier]

AUTHOR

@benoit-girard: Done.

[otizonaizit]

EDITOR

@benoit-girard:

I checked (superficially) that the provided code does not appear to simply write a predetermined set of values in log files, but does indeed implement the model. How deep should this check be?

Well, there is no point in checking the code line-by-line, but it would be awesome if you could have a general look at the code and verify that it makes sense and the implementation does look sound. Reproducing the figures and the data is just the first step, citing from the Editor Guidelines: the clarity of the code is an important criterion. Uncommented or obfuscated code is as bad as no code at all :)

[benoit-girard]

REVIEWER 2

@otizonaizit @rougier

I have read the code, it is clear, commented and appears to be a honest replication of the original model.

The modifications made to the code solved the identified problems.

I therefore recommend paper acceptance.

[MehdiKhamassi]

REVIEWER 1

Dear @otizonaizit @rougier,
I have now finished to review this paper and the code of the simulated model. Sorry for the delay. As initially said, I could not dedicate significant amount of time for this review before mid-July. I nevertheless tried to do it before, as asked by the Editor, but ran through technical problems which did not allow me to complete the review in the very limited time I had before mid-July. So I will also here write a few comments about the technical issues I ran through, hoping that this could help for future reviews.

First important thing, here is my review on the article:

REVIEW OF THE SUBMITTED ARTICLE

In the following, I define:

• "submitted article", "submitting authors", "submitted code" as the paper by Topalidou and Rougier currently under review at ReScience, and the related python code that they attached to the submission.
• "original article", "original authors", "original code" as the paper by Guthrie et al. 2013 in Journal of Neurophysiology, and the Delphi code for their model which they provided to the submitting authors according to statements in the submitted article.

I have read the submitted code. It is readable, commented. It seems to me that it honestly re-implements the model of the original article for real (in the sense that it does not look like a code written to cheat by specifically rebuilding the figures of the original article without instantiating the model proposed by the original authors) . The obtained figures are qualitatively reproducing figures 4 & 5 of the original article. So the submitted code of the model is acceptable as it is.

However, the text of the submitted paper should be improved a little bit before acceptance.

(Page 1, Methods) "We found some factual errors in the original article that have been corrected in this implementation."
What do you mean precisely? Were these errors in the description of the model made in the original article, although the original code for the model was correct? Or errors in the original code for the model although the description in the original article was accurate compared to how the model should have been, or corresponding to a slightly different version of the original code which had been previously used to generate the figures of the original article but which did not correspond to the code provided by the original authors?
Also, were these errors in terms of wrong parameter values? Errors in the equations as they are written in the original article? Something else?
If these were errors in the original article (and not in the original code), please dedicate a paragraph in the submitted article describing how the original article should be corrected. For instance, "the value of the noise parameter for the striatum in Table 1 of the original article should be 0.1". Other example: "Equation 6 in the original article should be rewritten: ...".

(Page 1, Methods) I think that the tables of the submitted article should include at least as much information as the tables in the original article. They could contain more information but not less. Here I do not see the values for the synaptic decay parameter (tau) which is shown in Table 6 of the original article. Could the submitting authors include it and double-check that the tables in the submitted article do include at least all parameters that have been described in tables of the original article?

(Page 4, Conclusion) Shouldn't the conclusion explicitly mention that a few equations or parameters from the model had to be modified from the original article in order to reproduce the same qualitative results?

At the very beginning of the article, after “A reference implementation for …”, please add the following information (Currently, part of this information is written in the text of the submitted article, which is not easy to find for the reader. This should be visible and directly accessible information for the reader, I think):

• Whether the original code of the model written by the original authors has been accessed to by the submitting authors
• If yes, whether the original code is accessible online or whether it was privately given by the original authors to the submitting authors? In the former case, please provide the URL to access the original code.
• Also in the case where the original code was accessed to by the submitting authors, please write in which language the original code was written. (Because the editor and the reviewers may take a different decision about the submitted article if the original code was already written in python and is already accessible online. They may decide either that a re-implementation is not acceptable for publication in ReScience. They may alternatively decide that it is acceptable, but that the reviewers should verify that the submitted code is a real re-implementation and not a clone of the original code. See below some raised general questions for ReScience in relation to this issue).

Is an article submitted to ReScience acceptable if it only reproduces a subpart of the figures of the original article, or even a single figure? Shouldn't a justification in this case be explicitly required in the submission guidelines of the journal? In the submitted article, it is written that figures 4 & 5 of the original article are "the main results" of the original article, implicitly meaning that the other figures are secondary and do not need to be reproduced in the submission to ReScience.
In such a case of partial reproduction of the results of the original article, shouldn't the submitted article include guidance, tips and suggestions to help a reader use the submitted code to reproduce additional results not included in the submitted article?

TYPOS:

(Page 1) "We’ve been unable to compile this original implementation but we’re able to run the provided Windows executable.". Abbreviations should be avoided in research artices. This should read: We have been unable [...] but we were able to ...

(Page 4 Results) "We did not reproduce all analysis of the original article but concentrate our efforts on". Should read "all analyses" and "concentrated".

---

EDITOR-IN-CHIEF

@MehdiKhamassi

I opened a new issue at ReScience/ReScience#7 such that you can post your global questions and issues you raised in your review. Thank you very much for such useful interaction.

I thereby removed the end of your review since you have posted it to the aforementioned issue.

[rougier]

AUTHOR

@MehdiKhamassi

We corrected the manuscript to take your comments into account. Please find below a detail of what has been made:

• Explanations of factual errors in the original article have been inserted in the manuscript
• All parameters are now present (tau and sigmoid parameters were missing)
• A new paragraph has been inserted concerning the learning rule that has been substantially modified in our implementation
• Introduction has been changed to explain we had access to the original Delphi sources. Unfortunately, thoses sources are not available online.
• Typos have been fixed.

[MehdiKhamassi]

REVIEWER 1

@otizonaizit

The authors have changed and improved the paper following my comments.
I therefore recommend paper acceptance.

[otizonaizit]

EDITOR

The authors successfully addressed all comments by reviewer 1 and reviewer 2. I hereby accept the submitter paper for publication.

@rougier Please do not modify your repository until further notice.

[otizonaizit]

EDITOR

This submission has been accepted for publication, and has been published at https://github.com/ReScience/ReScience/wiki/Current-Issue