Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Review Request : M. Topalidou, N. Rougier #3
I request a review for the reproduction of the following paper:
I believed the original results have been faithfully reproduced as explained in the accompanying article.
The repository lives at https://github.com/rougier/ReScience-submission/tree/topalidou-rougier
changed the title from
Review Request : M. Topalidou, N. Rougier
Jun 9, 2015
@rougier: I am going to edit this submission, thanks!
when I run the single-trial.py, the cortical activity does not go back to rest levels of activity after trial stop (after 2.5s), contrarily to what is reported in fig1.
The 250-trial test gives results similar to what is presented in the paper, however the program ends abnormally :
sys has not been imported where it should have been?
Well, there is no point in checking the code line-by-line, but it would be awesome if you could have a general look at the code and verify that it makes sense and the implementation does look sound. Reproducing the figures and the data is just the first step, citing from the Editor Guidelines: the clarity of the code is an important criterion. Uncommented or obfuscated code is as bad as no code at all :)
referenced this pull request
Jul 15, 2015
Dear @otizonaizit @rougier,
First important thing, here is my review on the article:
REVIEW OF THE SUBMITTED ARTICLE
In the following, I define:
I have read the submitted code. It is readable, commented. It seems to me that it honestly re-implements the model of the original article for real (in the sense that it does not look like a code written to cheat by specifically rebuilding the figures of the original article without instantiating the model proposed by the original authors) . The obtained figures are qualitatively reproducing figures 4 & 5 of the original article. So the submitted code of the model is acceptable as it is.
However, the text of the submitted paper should be improved a little bit before acceptance.
(Page 1, Methods) "We found some factual errors in the original article that have been corrected in this implementation."
(Page 1, Methods) I think that the tables of the submitted article should include at least as much information as the tables in the original article. They could contain more information but not less. Here I do not see the values for the synaptic decay parameter (tau) which is shown in Table 6 of the original article. Could the submitting authors include it and double-check that the tables in the submitted article do include at least all parameters that have been described in tables of the original article?
(Page 4, Conclusion) Shouldn't the conclusion explicitly mention that a few equations or parameters from the model had to be modified from the original article in order to reproduce the same qualitative results?
At the very beginning of the article, after “A reference implementation for …”, please add the following information (Currently, part of this information is written in the text of the submitted article, which is not easy to find for the reader. This should be visible and directly accessible information for the reader, I think):
Is an article submitted to ReScience acceptable if it only reproduces a subpart of the figures of the original article, or even a single figure? Shouldn't a justification in this case be explicitly required in the submission guidelines of the journal? In the submitted article, it is written that figures 4 & 5 of the original article are "the main results" of the original article, implicitly meaning that the other figures are secondary and do not need to be reproduced in the submission to ReScience.
(Page 1) "We’ve been unable to compile this original implementation but we’re able to run the provided Windows executable.". Abbreviations should be avoided in research artices. This should read: We have been unable [...] but we were able to ...
(Page 4 Results) "We did not reproduce all analysis of the original article but concentrate our efforts on". Should read "all analyses" and "concentrated".
I opened a new issue at ReScience/ReScience#7 such that you can post your global questions and issues you raised in your review. Thank you very much for such useful interaction.
I thereby removed the end of your review since you have posted it to the aforementioned issue.
We corrected the manuscript to take your comments into account. Please find below a detail of what has been made:
This submission has been accepted for publication, and has been published at https://github.com/ReScience/ReScience/wiki/Current-Issue