Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Review Request: Rostami, Ito, Denker, Grün #24
I request a review for the following replication:
Title: Spike Synchronization and Rate Modulation Differentially Involved in Motor Cortical Function
Author(s): Vahid Rostami, Junji Ito, Michael Denker and Sonja Grün
Thank you for your consideration,
changed the title from
V rostami j ito m denker s gruen 2016
Oct 28, 2016
Dear @Vahidrostami, I just realized that one of the new author (Sonja Grün) is also one of the author of the original article. Could you tell me a bit more about that. It's ok since the paper is alsmost 20 years old but I wonder what langage was the original implementation and if the new implementation is a brand new one or a simple adaptation.
Dear @rougier, The current implementation of the Unitary Events analysis method, which is the basis for the reproduced findings, is a completely new reimplementation of the analysis method in Python written by author VR (myself). This reimplementation was done from scratch without access to the original codes.
As we detail in the manuscript, tracing the provenance of the original figures was a major part of our reproduction. It turned out, that (i) two different analysis codes (one in IDL, the other in Matlab) were used for the original publication and (ii) that they are either no longer available (IDL version), or only available in a version that was heavily modified over the years to include additional features, however, without version control or unit testing (Matlab version). Thus, the reimplementation was a necessity. For maximum benefit for the community, the method was integrated as part of a recently established resource for tools for neuronal activity data analysis, the Electrophysiology Analysis Toolkit (Elephant). In the manuscript, we recount the detailed process of tracing what was done in the original study, which was a necessary step to arrive at the final reproduced figures using the reimplemented method.
In the meantime, you need to update the README.md file in the code directory to explain requirements and dependencies. And you need to add a LICENSE.txt in the code directory.
@rougier I just updated:
Please take a look and let me know if something is missing. Thanks
In this paper, authors have reproduced the results of . They re-implement the
The text is quite descriptive. The main algorithm is described in both text and pseudocode. The main tools that the authors use in their implementation are explicitly provided in the text (their implementation is based on Neo and Elephant) and the parameters are given as well.
The authors compared their results with the original ones by visual inspection due to the lack of the original data. One possible way to resolve this issue is to use computer vision methods in order to digitize the original images and extract the data from there and/or use similarity techniques to identify similarities between original and new results.
Axes in figure 3 are missing.
They include a License and a short description of the dependencies for running the code. However, there are some issues with the execution of the code. I ran the code as a standalone Python script and when I ran it using Python 2.7, the code is executed smoothly out-of-the-box.
There seems to be a misunderstanding regarding the reproduction of the results. We were able to exactly reproduce the original results and not only “close to the original ones” as stated here. Obviously this did not become clear in the submitted version. Therefore we changed the revised manuscript in two respects: 1) we included a direct quantitative comparison of the original results and the reproduced results and included a new figure (Figure 3) showing this (please also refer to our response to a comment below). 2) We improved the first paragraph of the Conclusion section to make the exact reproduction clear.
As suggested by the reviewers we now include a thorough comparison of the similarities between the original and the reproduced results in form of a new figure (Figure 3) in the revised manuscript. For doing so, we extracted the times of the spikes and the unitary events from the vector graphic image (PDF) in the original paper. Then, we directly compare these times to the times of the spikes and unitary events in our reproduced results by plotting the former against the latter. This procedure is explained in the revised manuscript in the end of the second paragraph of page 10.
"Axes in figure 3 are missing."
We agree that compatibility with Python 3 is missing in our code. The revised version of the code is now compatible with Python 3.
"2. If you try to run the code with method=surrogate_TrialByTrial it fails (is there any extra change that has to be done?)."
The failure was due to a bug in the unitary event function in Elephant. We thank the reviewer to have spotted this bug. We have fixed this bug and now the code with method=surrogate_TrialByTrial can be run (using the new version of Elephant).
"3. In addition, that would be nice if the code is provided as a standalone script along with the already existed notebook."
We agree with the reviewer’s suggestion and we added a standalone python script “Plotting_Figures.py” in the code repository.
"4. Furthermore, the utils.py script should be more documented (at least what the functions do and what the arguments of the functions are) and please put some comments about the plot_params_default dictionary."
We agree that the script was not documented very well. Thus we added more documentation.
"5. [optional] Verify that the code follows the PEP 8 (the code will be more readable)"
To increase the readability, as the reviewer suggested, we checked the code with pylint on emacs and changed the code to follow the PEP 8.
"6. Add the BSD license in the utils.py file as well."
Based on the reviewer suggestion, we added a license for all the scripts in the code repository.
"The authors of this replication did a remarkable, and difficult, job of reproducing the result of a 19-year-old article, whose data and code was only partially available. They explain in great detail the methodology they followed and the hurdles they faced, which offers an interesting window into the difficulty of reproducing results in neuroscience. Rather than producing code tailored to this replication, they included most of the code in an open-source scientific Python toolbox, Elephant, which potentially makes replicating other works on Unitary Events easier. At the end of the article, the authors conclude on the methodological challenges facing the handling and preservation of electrophysiological data."
We thank the reviewer for his positive comment.
" Main Remarks
The role of Sonja Grün in the original publication was that she developed the analysis method (the Unitary Events analysis method), implemented the method in IDL which was used by Alexa Riehle (first author of the original publication), and did all the testing of the method. She then re-implemented the method together with Markus Diesmann in Matlab. This version was used for the analysis of Figure 2.
"The efforts that were made to validate the results remain vague: “we compare each of our figures visually in detail with the original figures”. What does “in detail” means? Where a random sample of points compared? What about overlaying both result images (with different point color, one made transparent, so that points present in both have a distinct color) ? Nevertheless, on my non-exhaustive, non-random, visual inspection, results do appear completely identical."
As suggested by the reviewers we now include a thorough comparison of the similarities between the original and the reproduced results in form of a new figure (Figure 3) in the revised manuscript. For doing so, we extracted the times of the spikes and the unitary events from the vector graphic image (pdf) in the original paper. Then, we directly compare these times to the times of the spikes and unitary events in our reproduced results by plotting the former against the latter as well as examining the distribution of the differences between them. This procedure is explained in the revised manuscript in the end of the second paragraph of page 10. This replaces the validation by visual inspection with a quantitative measure of similarity of figures.
"Given the work that has been done, replicating figure 3 and figure 4b would seem to require comparatively little work. While the authors did show that they correctly implemented the UE method used in the original article, replicating figure 3 would show that no mistakes were made by in the original article, which is one of the primary goals of a replication."
We agree with the reviewer that it would be nice to have had replicated all figures of the original publication. We got back to the first author (AR) who generated Figures 3 and Figure 4b for the original publication. Figure 4b is a meta analysis over a large set of pairs of neurons recorded in many sessions. However, AR neither has the list of neuron pairs included in the analysis anymore nor the data sets. A similar answer we got for the data in Figure 3: the two data sets are still available, however, in a format / version where the trial markers for indicating the respective behavior for sorting the data into different classes (for A: the switch release, for B: correct / non-correct behavior) are not available anymore. For these reasons we were not able to replicate these figures.
" Other Remarks
We agree with this remark. We included this point in the main text as the general motivation for a reimplementation, and restated the justification as the requirement that applies to our particular case of reimplementation.
"In the first paragraph of page 2, the end of the paragraph is difficult to understand."
We agree that this sentence was not very clear. We found out that this sentence was not directly related to the current work. Therefore we decided to delete the sentence.
"The - 1 and the - N terms in 2^N - N - 1 should be explicitly explained (i.e. not zero, and not only one spike)."
According to the suggestion, we revised the corresponding sentence in the Methods section so that each of the terms is explicitly explained.
"There are instructions to install the code, but none on how to run it. It would be good for a person unfamiliar with jupyter notebooks to be able to know what to do."
We thank the reviewer for pointing out that the information was missing. We have added a reference to the documentation of the jupyter notebook in the README.md file of the main repository.
"With a notebook as the main vector of the reproduction, a http://mybinder.org/ link would make the reproduction accessible to anyone without installation."
In our response to a comment by the other reviewer, we have added a standalone python script for the reproduction of the two figures of the original article. Thus in the revised version of our pull request the main vector of the reproduction is the standalone script and hence those readers who are not used to Jupyter notebook can access the reproduction through the standalone script.
"Since the elaboration of the code, neo-0.4.0 (and neo-0.4.1) has been released. Is a git checkout still necessary? A requirements.txt file would simplify the installation of dependencies."
We thank the reviewer for pointing this out. We corrected the requirements in the jupyter notebook and we added a requirements.txt file in the code repository.
"p. 2: “the existing code” -> explain that this is the code of the original article."
We agree that our expression “existing code” was not very helpful. We anyhow had to rephrase the sentence and thereby removed this ambiguous expression.
"The plotting code is not compatible with Python 3. At minima, this should be clearly explained in the installation instructions, and print statements could all be provided parentheses. The exec syntax that remains incompatible with Python 3 is non-intuitive. The parameters could be accessed with a dictionary syntax (params[‘name’]) rather than injected into the current scope, which makes them indistinguishable from local variables, and makes the code more difficult to read."
The revised version of the code is now compatible with Python 3.
" Language Remarks
Thanks for spotting these typos, we corrected them.
"In the Pull Request
We thank the reviewer for the careful reading. We have corrected all the typos.
Authors addressed all of my comments. The source code is now commented and running out-of-the-box (bugs have been fixed). Authors have compared their results with the original ones using a more rigorous method. I recommend acceptance of the current version. Below I provide some minor comments regarding mostly some typos.
The authors have done a remarkable job of addressing my comments. I particularly appreciate the quantitative analysis provided by the new Figure 3. So much that I have a couple of remarks about it:
Figure 3 not reproduced
Ease of use
By the way, I assume you meant Elephant>=0.4.0, not 0.5.0 which is not released yet (latest release is 0.4.1, README.md is affected as well) ?
File "<ipython-input-5-90fba530d061>", line 16 print 'calculating UE ...' ^ SyntaxError: Missing parentheses in call to 'print'
Beyond those remarks, I fully recommend accepting the contribution.
Following a recent discussion (ReScience/ReScience#43) about the non-reproducibility of articles published in ReScience, I did a quick test on this one using the accepted but not yet published code. As explained in the earlier discussion, all I checked is if I can install the dependencies and run the software - I don't look at the results, since I am not competent in judging them anyway.
First remark: the instructions quote version numbers for everything except Python itself. In particular, it isn't clear if I should use Python 2 or Python 3.
Using Python 3.5 under macOS 10.11, I installed the dependencies via pip and ran
If that warning is expected and harmless, it would be nice to mention this in the instructions.
@rougier This is expected. This happens because the array contains nan when it tests for greater_equal. This warning occurs with python 3 not python 2. As we wrote the whole code in python 2 this warning slipped our attention. However as the reviewer suggested we are making the code compatible with Python 3.
@khinsen Thanks for your test. In the final version we will take care of this warning.
Figure 3 not reproduced
Regarding these two points, we are happy to share our code for extracting plot points from a PDF file, but we do not think that it should be included into the publication, for the following reason.
Ease of use
@rougier It clearly goes way beyond this particular review, and IANAL, and the risks are possibly not worth taking, but this may fall under Fair Use (http://fairuse.stanford.edu/overview/fair-use/what-is-fair-use/). Transformative, partial, with the intent to comment, and "the public reaps benefits from your review".
Also, we could ask.