Review Request: Rostami, Ito, Denker, Grün #24

Closed
wants to merge 27 commits into
from

Conversation

Projects
None yet
7 participants
@Vahidrostami

Vahidrostami commented Oct 28, 2016

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: Spike Synchronization and Rate Modulation Differentially Involved in Motor Cortical Function
Author(s): Alexa Riehle, , Sonja Grün, Markus Diesmann, and Ad Aertsen
Journal (or Conference): Science
Year: 1997
DOI: 10.1126/science.278.5345.1950
PDF: http://science.sciencemag.org/content/278/5345/1950

Replication

Author(s): Vahid Rostami, Junji Ito, Michael Denker and Sonja Grün
Repository: https://github.com/Vahidrostami/ReScience-submission/tree/VRostami-JIto-MDenker-SGruen-2016
PDF: https://github.com/Vahidrostami/ReScience-submission/blob/VRostami-JIto-MDenker-SGruen-2016/article/VRostami_JIto_MDenker_SGruen_2016.pdf
Keywords: Spike time synchrony, Unitary Events method , Python
Language: English
Domain: Life Science

Results

  • Article has been fully replicated
  • Article has been partially replicated
  • Article has not been replicated

Thank you for your consideration,
Vahid Rostami

Potential reviewers


EDITOR

  • Editor acknowledgment (@rougier) October 31, 2016
  • Reviewer 1 (@gdetor) November 8, 2016
  • Reviewer 2 (@benureau) November 10, 2016
  • Review 1 decision [accept] April 6, 2017
  • Review 2 decision [accept] April 19, 2017
  • Editor decision [accept] April 19, 2017

@Vahidrostami Vahidrostami changed the title from V rostami j ito m denker s gruen 2016 to Review Request Oct 28, 2016

@Vahidrostami Vahidrostami changed the title from Review Request to Review Request: Vrostami, Jito, Mdenker, Sgrün Oct 28, 2016

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Oct 28, 2016

Member

Thanks for your submission. An editor will soon be assigned.

Member

rougier commented Oct 28, 2016

Thanks for your submission. An editor will soon be assigned.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Oct 30, 2016

Member

Dear @Vahidrostami, I just realized that one of the new author (Sonja Grün) is also one of the author of the original article. Could you tell me a bit more about that. It's ok since the paper is alsmost 20 years old but I wonder what langage was the original implementation and if the new implementation is a brand new one or a simple adaptation.

Member

rougier commented Oct 30, 2016

Dear @Vahidrostami, I just realized that one of the new author (Sonja Grün) is also one of the author of the original article. Could you tell me a bit more about that. It's ok since the paper is alsmost 20 years old but I wonder what langage was the original implementation and if the new implementation is a brand new one or a simple adaptation.

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Oct 31, 2016

Dear @rougier, The current implementation of the Unitary Events analysis method, which is the basis for the reproduced findings, is a completely new reimplementation of the analysis method in Python written by author VR (myself). This reimplementation was done from scratch without access to the original codes.

As we detail in the manuscript, tracing the provenance of the original figures was a major part of our reproduction. It turned out, that (i) two different analysis codes (one in IDL, the other in Matlab) were used for the original publication and (ii) that they are either no longer available (IDL version), or only available in a version that was heavily modified over the years to include additional features, however, without version control or unit testing (Matlab version). Thus, the reimplementation was a necessity. For maximum benefit for the community, the method was integrated as part of a recently established resource for tools for neuronal activity data analysis, the Electrophysiology Analysis Toolkit (Elephant). In the manuscript, we recount the detailed process of tracing what was done in the original study, which was a necessary step to arrive at the final reproduced figures using the reimplemented method.

Vahidrostami commented Oct 31, 2016

Dear @rougier, The current implementation of the Unitary Events analysis method, which is the basis for the reproduced findings, is a completely new reimplementation of the analysis method in Python written by author VR (myself). This reimplementation was done from scratch without access to the original codes.

As we detail in the manuscript, tracing the provenance of the original figures was a major part of our reproduction. It turned out, that (i) two different analysis codes (one in IDL, the other in Matlab) were used for the original publication and (ii) that they are either no longer available (IDL version), or only available in a version that was heavily modified over the years to include additional features, however, without version control or unit testing (Matlab version). Thus, the reimplementation was a necessity. For maximum benefit for the community, the method was integrated as part of a recently established resource for tools for neuronal activity data analysis, the Electrophysiology Analysis Toolkit (Elephant). In the manuscript, we recount the detailed process of tracing what was done in the original study, which was a necessary step to arrive at the final reproduced figures using the reimplemented method.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Oct 31, 2016

Member

Thank you for the detailed explanation. In the meantime, I've started to read the paper where it is also nicely explained.

Member

rougier commented Oct 31, 2016

Thank you for the detailed explanation. In the meantime, I've started to read the paper where it is also nicely explained.

@rougier rougier self-assigned this Oct 31, 2016

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Oct 31, 2016

Member

@Vahidrostami I will edit your submission and will soon assign two reviewers.

Member

rougier commented Oct 31, 2016

@Vahidrostami I will edit your submission and will soon assign two reviewers.

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Oct 31, 2016

@rougier, Thank you very much for the quick reply.

@rougier, Thank you very much for the quick reply.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Oct 31, 2016

Member

In the meantime, you need to update the README.md file in the code directory to explain requirements and dependencies. And you need to add a LICENSE.txt in the code directory.
For the root directory, you can also update the README.md to explain what this repository is about (you can check for example https://github.com/ReScience-Archives/Vitay-2016)

Member

rougier commented Oct 31, 2016

In the meantime, you need to update the README.md file in the code directory to explain requirements and dependencies. And you need to add a LICENSE.txt in the code directory.
For the root directory, you can also update the README.md to explain what this repository is about (you can check for example https://github.com/ReScience-Archives/Vitay-2016)

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Oct 31, 2016

Ok. I will update the repository of our submission in the upcoming days.

Ok. I will update the repository of our submission in the upcoming days.

@rougier rougier changed the title from Review Request: Vrostami, Jito, Mdenker, Sgrün to Review Request: Vrostami, Jito, Mdenker, Grün Nov 1, 2016

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 2, 2016

Member

@rossant @apdavison Can you review this submission ?

Member

rougier commented Nov 2, 2016

@rossant @apdavison Can you review this submission ?

@apdavison

This comment has been minimized.

Show comment
Hide comment
@apdavison

apdavison Nov 2, 2016

I have an existing collaboration with Sonja Grün; I also contribute to the Elephant toolbox mentioned in the article. Perhaps this is too much of a conflict of interest?

I have an existing collaboration with Sonja Grün; I also contribute to the Elephant toolbox mentioned in the article. Perhaps this is too much of a conflict of interest?

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 2, 2016

Member

@apdavison Thank you for the quick answer and yes, the existing collaboration might be a problem. Would you have some name to suggest (someone familiar with Elephant and Neo) by any chance ?

Member

rougier commented Nov 2, 2016

@apdavison Thank you for the quick answer and yes, the existing collaboration might be a problem. Would you have some name to suggest (someone familiar with Elephant and Neo) by any chance ?

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 3, 2016

Member

@benoit-girard Could you review this submission ?

Member

rougier commented Nov 3, 2016

@benoit-girard Could you review this submission ?

@rougier rougier changed the title from Review Request: Vrostami, Jito, Mdenker, Grün to Review Request: Rostami, Ito, Denker, Grün Nov 4, 2016

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Nov 4, 2016

@rougier I just updated:

  1. the code repository by adding a Licence for our code and explaining the requirements and dependencies.
  2. the README.md in the root directory to explain what this repository is about.

Please take a look and let me know if something is missing. Thanks

@rougier I just updated:

  1. the code repository by adding a Licence for our code and explaining the requirements and dependencies.
  2. the README.md in the root directory to explain what this repository is about.

Please take a look and let me know if something is missing. Thanks

@rossant

This comment has been minimized.

Show comment
Hide comment
@rossant

rossant Nov 7, 2016

@rougier unfortunately I don't have time to do a full review, although I could do a quick code review if that's useful...

rossant commented Nov 7, 2016

@rougier unfortunately I don't have time to do a full review, although I could do a quick code review if that's useful...

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 7, 2016

Member

@rossant Thanks for the answer.

Member

rougier commented Nov 7, 2016

@rossant Thanks for the answer.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 7, 2016

Member

@gdetor Could you handle this review ?

Member

rougier commented Nov 7, 2016

@gdetor Could you handle this review ?

@gdetor

This comment has been minimized.

Show comment
Hide comment

gdetor commented Nov 7, 2016

@rougier Yes.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 8, 2016

Member

@gdetor Thank you.

@benureau Could you handle this review as well ?

Member

rougier commented Nov 8, 2016

@gdetor Thank you.

@benureau Could you handle this review as well ?

@benureau

This comment has been minimized.

Show comment
Hide comment
@benureau

benureau Nov 10, 2016

@rougier Yes. I expect it will be done in two weeks to one month.

@rougier Yes. I expect it will be done in two weeks to one month.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 10, 2016

Member

Ok, thanks @benureau

Member

rougier commented Nov 10, 2016

Ok, thanks @benureau

@rougier rougier added the 02 - Review label Nov 10, 2016

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Nov 17, 2016

GENERAL

In this paper, authors have reproduced the results of [26]. They re-implement the
Unitary Events (UE) method used in [26]. The UE method gets information regarding spike synchrony between neural activities. Authors managed to re-implement the
original method and get results similar to the original ones. They provide a brief
description of the original experiment rendering the reading and comprehension of the
paper easier. Finally, their results are close to the original ones. They are not matching
exactly due to some binning discrepancies between data. Below is my comments on the text and the code.

TEXT

The text is quite descriptive. The main algorithm is described in both text and pseudocode. The main tools that the authors use in their implementation are explicitly provided in the text (their implementation is based on Neo and Elephant) and the parameters are given as well.

The authors compared their results with the original ones by visual inspection due to the lack of the original data. One possible way to resolve this issue is to use computer vision methods in order to digitize the original images and extract the data from there and/or use similarity techniques to identify similarities between original and new results.

Axes in figure 3 are missing.

CODE

They include a License and a short description of the dependencies for running the code. However, there are some issues with the execution of the code. I ran the code as a standalone Python script and when I ran it using Python 2.7, the code is executed smoothly out-of-the-box.

  1. When I used Python 3, then the code terminates with a NameError. That’s because of the ‘exec’ function at lines 107 and 275 of utils.py file. In Python 2.7 exec is a statement, but in Python 3 it’s a function and the scope can be different.
  2. If you try to run the code with method=surrogate_TrialByTrial it fails (is there any extra change that has to be done?).
  3. In addition, that would be nice if the code is provided as a standalone script along with the already existed notebook.
  4. Furthermore, the utils.py script should be more documented (at least what the functions do and what the arguments of the functions are) and please put some comments about the plot_params_default dictionary.
  5. [optional] Verify that the code follows the PEP 8 (the code will be more readable)
  6. Add the BSD license in the utils.py file as well.

gdetor commented Nov 17, 2016

GENERAL

In this paper, authors have reproduced the results of [26]. They re-implement the
Unitary Events (UE) method used in [26]. The UE method gets information regarding spike synchrony between neural activities. Authors managed to re-implement the
original method and get results similar to the original ones. They provide a brief
description of the original experiment rendering the reading and comprehension of the
paper easier. Finally, their results are close to the original ones. They are not matching
exactly due to some binning discrepancies between data. Below is my comments on the text and the code.

TEXT

The text is quite descriptive. The main algorithm is described in both text and pseudocode. The main tools that the authors use in their implementation are explicitly provided in the text (their implementation is based on Neo and Elephant) and the parameters are given as well.

The authors compared their results with the original ones by visual inspection due to the lack of the original data. One possible way to resolve this issue is to use computer vision methods in order to digitize the original images and extract the data from there and/or use similarity techniques to identify similarities between original and new results.

Axes in figure 3 are missing.

CODE

They include a License and a short description of the dependencies for running the code. However, there are some issues with the execution of the code. I ran the code as a standalone Python script and when I ran it using Python 2.7, the code is executed smoothly out-of-the-box.

  1. When I used Python 3, then the code terminates with a NameError. That’s because of the ‘exec’ function at lines 107 and 275 of utils.py file. In Python 2.7 exec is a statement, but in Python 3 it’s a function and the scope can be different.
  2. If you try to run the code with method=surrogate_TrialByTrial it fails (is there any extra change that has to be done?).
  3. In addition, that would be nice if the code is provided as a standalone script along with the already existed notebook.
  4. Furthermore, the utils.py script should be more documented (at least what the functions do and what the arguments of the functions are) and please put some comments about the plot_params_default dictionary.
  5. [optional] Verify that the code follows the PEP 8 (the code will be more readable)
  6. Add the BSD license in the utils.py file as well.
@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Dec 12, 2016

Member

@benureau Any update on your side ?

Member

rougier commented Dec 12, 2016

@benureau Any update on your side ?

@rougier

This comment has been minimized.

Show comment
Hide comment
Member

rougier commented Dec 18, 2016

@benureau 🛎

@benureau

This comment has been minimized.

Show comment
Hide comment
@benureau

benureau Jan 3, 2017

I am a bit behind, I should be done with this review at the end of the week.

benureau commented Jan 3, 2017

I am a bit behind, I should be done with this review at the end of the week.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Feb 23, 2017

Member

@Vahidrostami What's the status of your revision ?

Member

rougier commented Feb 23, 2017

@Vahidrostami What's the status of your revision ?

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Feb 28, 2017

@rougier We are working on that. I expect that we can respond to the reviewers comments in 3 weeks.

@rougier We are working on that. I expect that we can respond to the reviewers comments in 3 weeks.

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Mar 29, 2017

@rougier
@gdetor Thanks a lot for your comments. Please find below a detailed response to your comments:

" General:
In this paper, authors have reproduced the results of [26]. They re-implement the
Unitary Events (UE) method used in [26]. The UE method gets information regarding spike synchrony between neural activities. Authors managed to re-implement the original method and get results similar to the original ones. They provide a brief description of the original experiment rendering the reading and comprehension of the paper easier. Finally, their results are close to the original ones. They are not matching exactly due to some binning discrepancies between data. Below is my comments on the text and the code."

There seems to be a misunderstanding regarding the reproduction of the results. We were able to exactly reproduce the original results and not only “close to the original ones” as stated here. Obviously this did not become clear in the submitted version. Therefore we changed the revised manuscript in two respects: 1) we included a direct quantitative comparison of the original results and the reproduced results and included a new figure (Figure 3) showing this (please also refer to our response to a comment below). 2) We improved the first paragraph of the Conclusion section to make the exact reproduction clear.

" Text
The text is quite descriptive. The main algorithm is described in both text and pseudocode. The main tools that the authors use in their implementation are explicitly provided in the text (their implementation is based on Neo and Elephant) and the parameters are given as well.
The authors compared their results with the original ones by visual inspection due to the lack of the original data. One possible way to resolve this issue is to use computer vision methods in order to digitize the original images and extract the data from there and/or use similarity techniques to identify similarities between original and new results."

As suggested by the reviewers we now include a thorough comparison of the similarities between the original and the reproduced results in form of a new figure (Figure 3) in the revised manuscript. For doing so, we extracted the times of the spikes and the unitary events from the vector graphic image (PDF) in the original paper. Then, we directly compare these times to the times of the spikes and unitary events in our reproduced results by plotting the former against the latter. This procedure is explained in the revised manuscript in the end of the second paragraph of page 10.

"Axes in figure 3 are missing."
We thank the reviewer for pointing this out. We have added axes labels to the mentioned figure.

" Code
They include a License and a short description of the dependencies for running the code. However, there are some issues with the execution of the code. I ran the code as a standalone Python script and when I ran it using Python 2.7, the code is executed smoothly out-of-the-box.

  1. When I used Python 3, then the code terminates with a NameError. That’s because of the ‘exec’ function at lines 107 and 275 of utils.py file. In Python 2.7 exec is a statement, but in Python 3 it’s a function and the scope can be different."

We agree that compatibility with Python 3 is missing in our code. The revised version of the code is now compatible with Python 3.

"2. If you try to run the code with method=surrogate_TrialByTrial it fails (is there any extra change that has to be done?)."

The failure was due to a bug in the unitary event function in Elephant. We thank the reviewer to have spotted this bug. We have fixed this bug and now the code with method=surrogate_TrialByTrial can be run (using the new version of Elephant).

"3. In addition, that would be nice if the code is provided as a standalone script along with the already existed notebook."

We agree with the reviewer’s suggestion and we added a standalone python script “Plotting_Figures.py” in the code repository.

"4. Furthermore, the utils.py script should be more documented (at least what the functions do and what the arguments of the functions are) and please put some comments about the plot_params_default dictionary."

We agree that the script was not documented very well. Thus we added more documentation.

"5. [optional] Verify that the code follows the PEP 8 (the code will be more readable)"

To increase the readability, as the reviewer suggested, we checked the code with pylint on emacs and changed the code to follow the PEP 8.

"6. Add the BSD license in the utils.py file as well."

Based on the reviewer suggestion, we added a license for all the scripts in the code repository.

Vahidrostami commented Mar 29, 2017

@rougier
@gdetor Thanks a lot for your comments. Please find below a detailed response to your comments:

" General:
In this paper, authors have reproduced the results of [26]. They re-implement the
Unitary Events (UE) method used in [26]. The UE method gets information regarding spike synchrony between neural activities. Authors managed to re-implement the original method and get results similar to the original ones. They provide a brief description of the original experiment rendering the reading and comprehension of the paper easier. Finally, their results are close to the original ones. They are not matching exactly due to some binning discrepancies between data. Below is my comments on the text and the code."

There seems to be a misunderstanding regarding the reproduction of the results. We were able to exactly reproduce the original results and not only “close to the original ones” as stated here. Obviously this did not become clear in the submitted version. Therefore we changed the revised manuscript in two respects: 1) we included a direct quantitative comparison of the original results and the reproduced results and included a new figure (Figure 3) showing this (please also refer to our response to a comment below). 2) We improved the first paragraph of the Conclusion section to make the exact reproduction clear.

" Text
The text is quite descriptive. The main algorithm is described in both text and pseudocode. The main tools that the authors use in their implementation are explicitly provided in the text (their implementation is based on Neo and Elephant) and the parameters are given as well.
The authors compared their results with the original ones by visual inspection due to the lack of the original data. One possible way to resolve this issue is to use computer vision methods in order to digitize the original images and extract the data from there and/or use similarity techniques to identify similarities between original and new results."

As suggested by the reviewers we now include a thorough comparison of the similarities between the original and the reproduced results in form of a new figure (Figure 3) in the revised manuscript. For doing so, we extracted the times of the spikes and the unitary events from the vector graphic image (PDF) in the original paper. Then, we directly compare these times to the times of the spikes and unitary events in our reproduced results by plotting the former against the latter. This procedure is explained in the revised manuscript in the end of the second paragraph of page 10.

"Axes in figure 3 are missing."
We thank the reviewer for pointing this out. We have added axes labels to the mentioned figure.

" Code
They include a License and a short description of the dependencies for running the code. However, there are some issues with the execution of the code. I ran the code as a standalone Python script and when I ran it using Python 2.7, the code is executed smoothly out-of-the-box.

  1. When I used Python 3, then the code terminates with a NameError. That’s because of the ‘exec’ function at lines 107 and 275 of utils.py file. In Python 2.7 exec is a statement, but in Python 3 it’s a function and the scope can be different."

We agree that compatibility with Python 3 is missing in our code. The revised version of the code is now compatible with Python 3.

"2. If you try to run the code with method=surrogate_TrialByTrial it fails (is there any extra change that has to be done?)."

The failure was due to a bug in the unitary event function in Elephant. We thank the reviewer to have spotted this bug. We have fixed this bug and now the code with method=surrogate_TrialByTrial can be run (using the new version of Elephant).

"3. In addition, that would be nice if the code is provided as a standalone script along with the already existed notebook."

We agree with the reviewer’s suggestion and we added a standalone python script “Plotting_Figures.py” in the code repository.

"4. Furthermore, the utils.py script should be more documented (at least what the functions do and what the arguments of the functions are) and please put some comments about the plot_params_default dictionary."

We agree that the script was not documented very well. Thus we added more documentation.

"5. [optional] Verify that the code follows the PEP 8 (the code will be more readable)"

To increase the readability, as the reviewer suggested, we checked the code with pylint on emacs and changed the code to follow the PEP 8.

"6. Add the BSD license in the utils.py file as well."

Based on the reviewer suggestion, we added a license for all the scripts in the code repository.

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Mar 29, 2017

@rougier
@benureau Thanks a lot for your comments. Please find below a detailed response to your comments:

"The authors of this replication did a remarkable, and difficult, job of reproducing the result of a 19-year-old article, whose data and code was only partially available. They explain in great detail the methodology they followed and the hurdles they faced, which offers an interesting window into the difficulty of reproducing results in neuroscience. Rather than producing code tailored to this replication, they included most of the code in an open-source scientific Python toolbox, Elephant, which potentially makes replicating other works on Unitary Events easier. At the end of the article, the authors conclude on the methodological challenges facing the handling and preservation of electrophysiological data."

We thank the reviewer for his positive comment.

" Main Remarks
The presence in this replication of an author of the original paper, Sonja Grün, should be addressed at some point in the text. What was her role in the publication, in relationship to the work she did for the original article?"

The role of Sonja Grün in the original publication was that she developed the analysis method (the Unitary Events analysis method), implemented the method in IDL which was used by Alexa Riehle (first author of the original publication), and did all the testing of the method. She then re-implemented the method together with Markus Diesmann in Matlab. This version was used for the analysis of Figure 2.
Her role in the current manuscript was that she initiated the contact to Markus Diesmann and Alexa Riehle, that she remembered that there was a difference in the results dependent on the trial alignment, and she indicated to potential solutions to several of the problems that arose during the reproduction of the results.
To clarify the contributions of individual authors, we include in the new version of the manuscript the author contributions.

"The efforts that were made to validate the results remain vague: “we compare each of our figures visually in detail with the original figures”. What does “in detail” means? Where a random sample of points compared? What about overlaying both result images (with different point color, one made transparent, so that points present in both have a distinct color) ? Nevertheless, on my non-exhaustive, non-random, visual inspection, results do appear completely identical."

As suggested by the reviewers we now include a thorough comparison of the similarities between the original and the reproduced results in form of a new figure (Figure 3) in the revised manuscript. For doing so, we extracted the times of the spikes and the unitary events from the vector graphic image (pdf) in the original paper. Then, we directly compare these times to the times of the spikes and unitary events in our reproduced results by plotting the former against the latter as well as examining the distribution of the differences between them. This procedure is explained in the revised manuscript in the end of the second paragraph of page 10. This replaces the validation by visual inspection with a quantitative measure of similarity of figures.

"Given the work that has been done, replicating figure 3 and figure 4b would seem to require comparatively little work. While the authors did show that they correctly implemented the UE method used in the original article, replicating figure 3 would show that no mistakes were made by in the original article, which is one of the primary goals of a replication."

We agree with the reviewer that it would be nice to have had replicated all figures of the original publication. We got back to the first author (AR) who generated Figures 3 and Figure 4b for the original publication. Figure 4b is a meta analysis over a large set of pairs of neurons recorded in many sessions. However, AR neither has the list of neuron pairs included in the analysis anymore nor the data sets. A similar answer we got for the data in Figure 3: the two data sets are still available, however, in a format / version where the trial markers for indicating the respective behavior for sorting the data into different classes (for A: the switch release, for B: correct / non-correct behavior) are not available anymore. For these reasons we were not able to replicate these figures.

" Other Remarks
The justification for the replication of the existing article p. 2 is not properly introduced: in the context of scientific replication, a reimplementation is always interesting, to reduce the risk that the results are due to small implementation details or mistakes."

We agree with this remark. We included this point in the main text as the general motivation for a reimplementation, and restated the justification as the requirement that applies to our particular case of reimplementation.

"In the first paragraph of page 2, the end of the paragraph is difficult to understand."

We agree that this sentence was not very clear. We found out that this sentence was not directly related to the current work. Therefore we decided to delete the sentence.

"The - 1 and the - N terms in 2^N - N - 1 should be explicitly explained (i.e. not zero, and not only one spike)."

According to the suggestion, we revised the corresponding sentence in the Methods section so that each of the terms is explicitly explained.

"There are instructions to install the code, but none on how to run it. It would be good for a person unfamiliar with jupyter notebooks to be able to know what to do."

We thank the reviewer for pointing out that the information was missing. We have added a reference to the documentation of the jupyter notebook in the README.md file of the main repository.

"With a notebook as the main vector of the reproduction, a http://mybinder.org/ link would make the reproduction accessible to anyone without installation."

In our response to a comment by the other reviewer, we have added a standalone python script for the reproduction of the two figures of the original article. Thus in the revised version of our pull request the main vector of the reproduction is the standalone script and hence those readers who are not used to Jupyter notebook can access the reproduction through the standalone script.

"Since the elaboration of the code, neo-0.4.0 (and neo-0.4.1) has been released. Is a git checkout still necessary? A requirements.txt file would simplify the installation of dependencies."

We thank the reviewer for pointing this out. We corrected the requirements in the jupyter notebook and we added a requirements.txt file in the code repository.

"p. 2: “the existing code” -> explain that this is the code of the original article."

We agree that our expression “existing code” was not very helpful. We anyhow had to rephrase the sentence and thereby removed this ambiguous expression.

"The plotting code is not compatible with Python 3. At minima, this should be clearly explained in the installation instructions, and print statements could all be provided parentheses. The exec syntax that remains incompatible with Python 3 is non-intuitive. The parameters could be accessed with a dictionary syntax (params[‘name’]) rather than injected into the current scope, which makes them indistinguishable from local variables, and makes the code more difficult to read."

The revised version of the code is now compatible with Python 3.

" Language Remarks
“a few ms” -> “a few milliseconds”
“also other features” -> “, other features”
“occurence” -> “occurrence”
“Acknowldegements” -> “Acknowledgements”
“Diescretize” -> “Discretize”
“coversion” -> “conversion”
p. 13: “the third author” -> “the third author, MD,” (for consistency)"

Thanks for spotting these typos, we corrected them.

"In the Pull Request
BinndedSpikeTrain -> BinnedSpikeTrain
whithin -> within
Qunatity -> Quantity"

We thank the reviewer for the careful reading. We have corrected all the typos.

@rougier
@benureau Thanks a lot for your comments. Please find below a detailed response to your comments:

"The authors of this replication did a remarkable, and difficult, job of reproducing the result of a 19-year-old article, whose data and code was only partially available. They explain in great detail the methodology they followed and the hurdles they faced, which offers an interesting window into the difficulty of reproducing results in neuroscience. Rather than producing code tailored to this replication, they included most of the code in an open-source scientific Python toolbox, Elephant, which potentially makes replicating other works on Unitary Events easier. At the end of the article, the authors conclude on the methodological challenges facing the handling and preservation of electrophysiological data."

We thank the reviewer for his positive comment.

" Main Remarks
The presence in this replication of an author of the original paper, Sonja Grün, should be addressed at some point in the text. What was her role in the publication, in relationship to the work she did for the original article?"

The role of Sonja Grün in the original publication was that she developed the analysis method (the Unitary Events analysis method), implemented the method in IDL which was used by Alexa Riehle (first author of the original publication), and did all the testing of the method. She then re-implemented the method together with Markus Diesmann in Matlab. This version was used for the analysis of Figure 2.
Her role in the current manuscript was that she initiated the contact to Markus Diesmann and Alexa Riehle, that she remembered that there was a difference in the results dependent on the trial alignment, and she indicated to potential solutions to several of the problems that arose during the reproduction of the results.
To clarify the contributions of individual authors, we include in the new version of the manuscript the author contributions.

"The efforts that were made to validate the results remain vague: “we compare each of our figures visually in detail with the original figures”. What does “in detail” means? Where a random sample of points compared? What about overlaying both result images (with different point color, one made transparent, so that points present in both have a distinct color) ? Nevertheless, on my non-exhaustive, non-random, visual inspection, results do appear completely identical."

As suggested by the reviewers we now include a thorough comparison of the similarities between the original and the reproduced results in form of a new figure (Figure 3) in the revised manuscript. For doing so, we extracted the times of the spikes and the unitary events from the vector graphic image (pdf) in the original paper. Then, we directly compare these times to the times of the spikes and unitary events in our reproduced results by plotting the former against the latter as well as examining the distribution of the differences between them. This procedure is explained in the revised manuscript in the end of the second paragraph of page 10. This replaces the validation by visual inspection with a quantitative measure of similarity of figures.

"Given the work that has been done, replicating figure 3 and figure 4b would seem to require comparatively little work. While the authors did show that they correctly implemented the UE method used in the original article, replicating figure 3 would show that no mistakes were made by in the original article, which is one of the primary goals of a replication."

We agree with the reviewer that it would be nice to have had replicated all figures of the original publication. We got back to the first author (AR) who generated Figures 3 and Figure 4b for the original publication. Figure 4b is a meta analysis over a large set of pairs of neurons recorded in many sessions. However, AR neither has the list of neuron pairs included in the analysis anymore nor the data sets. A similar answer we got for the data in Figure 3: the two data sets are still available, however, in a format / version where the trial markers for indicating the respective behavior for sorting the data into different classes (for A: the switch release, for B: correct / non-correct behavior) are not available anymore. For these reasons we were not able to replicate these figures.

" Other Remarks
The justification for the replication of the existing article p. 2 is not properly introduced: in the context of scientific replication, a reimplementation is always interesting, to reduce the risk that the results are due to small implementation details or mistakes."

We agree with this remark. We included this point in the main text as the general motivation for a reimplementation, and restated the justification as the requirement that applies to our particular case of reimplementation.

"In the first paragraph of page 2, the end of the paragraph is difficult to understand."

We agree that this sentence was not very clear. We found out that this sentence was not directly related to the current work. Therefore we decided to delete the sentence.

"The - 1 and the - N terms in 2^N - N - 1 should be explicitly explained (i.e. not zero, and not only one spike)."

According to the suggestion, we revised the corresponding sentence in the Methods section so that each of the terms is explicitly explained.

"There are instructions to install the code, but none on how to run it. It would be good for a person unfamiliar with jupyter notebooks to be able to know what to do."

We thank the reviewer for pointing out that the information was missing. We have added a reference to the documentation of the jupyter notebook in the README.md file of the main repository.

"With a notebook as the main vector of the reproduction, a http://mybinder.org/ link would make the reproduction accessible to anyone without installation."

In our response to a comment by the other reviewer, we have added a standalone python script for the reproduction of the two figures of the original article. Thus in the revised version of our pull request the main vector of the reproduction is the standalone script and hence those readers who are not used to Jupyter notebook can access the reproduction through the standalone script.

"Since the elaboration of the code, neo-0.4.0 (and neo-0.4.1) has been released. Is a git checkout still necessary? A requirements.txt file would simplify the installation of dependencies."

We thank the reviewer for pointing this out. We corrected the requirements in the jupyter notebook and we added a requirements.txt file in the code repository.

"p. 2: “the existing code” -> explain that this is the code of the original article."

We agree that our expression “existing code” was not very helpful. We anyhow had to rephrase the sentence and thereby removed this ambiguous expression.

"The plotting code is not compatible with Python 3. At minima, this should be clearly explained in the installation instructions, and print statements could all be provided parentheses. The exec syntax that remains incompatible with Python 3 is non-intuitive. The parameters could be accessed with a dictionary syntax (params[‘name’]) rather than injected into the current scope, which makes them indistinguishable from local variables, and makes the code more difficult to read."

The revised version of the code is now compatible with Python 3.

" Language Remarks
“a few ms” -> “a few milliseconds”
“also other features” -> “, other features”
“occurence” -> “occurrence”
“Acknowldegements” -> “Acknowledgements”
“Diescretize” -> “Discretize”
“coversion” -> “conversion”
p. 13: “the third author” -> “the third author, MD,” (for consistency)"

Thanks for spotting these typos, we corrected them.

"In the Pull Request
BinndedSpikeTrain -> BinnedSpikeTrain
whithin -> within
Qunatity -> Quantity"

We thank the reviewer for the careful reading. We have corrected all the typos.

@gdetor

This comment has been minimized.

Show comment
Hide comment
@gdetor

gdetor Apr 5, 2017

Authors addressed all of my comments. The source code is now commented and running out-of-the-box (bugs have been fixed). Authors have compared their results with the original ones using a more rigorous method. I recommend acceptance of the current version. Below I provide some minor comments regarding mostly some typos.

Minor comments

  1. page 7, Results section, first paragraph, first sentence: "the data still available -> since for the [...] available anymore" -> since for the rest of the figures data were either incomplete [...] or not available.
  2. page 7, Results section, first paragraph, last sentence: "arbitrary numbers of neurons" -> arbitrary number of neurons
  3. Page 10, Results section, third paragraph: "spike times correspond to to the" -> spike times correspond to the
  4. Page 12, Conclusion, first paragraph, third sentence: "This difference in ... in the results"-> This difference in the alignment would not affect the results
  5. Page 15, Conclusion, fifth paragraph, second sentence: "the reproduction is based the values"-> is based on the values
  6. Could you please provide an English version of Figure 5?

gdetor commented Apr 5, 2017

Authors addressed all of my comments. The source code is now commented and running out-of-the-box (bugs have been fixed). Authors have compared their results with the original ones using a more rigorous method. I recommend acceptance of the current version. Below I provide some minor comments regarding mostly some typos.

Minor comments

  1. page 7, Results section, first paragraph, first sentence: "the data still available -> since for the [...] available anymore" -> since for the rest of the figures data were either incomplete [...] or not available.
  2. page 7, Results section, first paragraph, last sentence: "arbitrary numbers of neurons" -> arbitrary number of neurons
  3. Page 10, Results section, third paragraph: "spike times correspond to to the" -> spike times correspond to the
  4. Page 12, Conclusion, first paragraph, third sentence: "This difference in ... in the results"-> This difference in the alignment would not affect the results
  5. Page 15, Conclusion, fifth paragraph, second sentence: "the reproduction is based the values"-> is based on the values
  6. Could you please provide an English version of Figure 5?
@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 6, 2017

Member

@gdetor Thanks.
@benureau Are you satisfied with the corrections?
@Vahidrostami Can you address the minor comments from @gdetor ?

Member

rougier commented Apr 6, 2017

@gdetor Thanks.
@benureau Are you satisfied with the corrections?
@Vahidrostami Can you address the minor comments from @gdetor ?

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Apr 13, 2017

@rougier
@gdetor Thanks a lot for the careful reading. I have fixed the typos and provided an English version of Figure 5.

@rougier
@gdetor Thanks a lot for the careful reading. I have fixed the typos and provided an English version of Figure 5.

@benureau

This comment has been minimized.

Show comment
Hide comment
@benureau

benureau Apr 19, 2017

The authors have done a remarkable job of addressing my comments. I particularly appreciate the quantitative analysis provided by the new Figure 3. So much that I have a couple of remarks about it:

Figure 3 not reproduced

  • The Plotting_figure.py code and the notebook talk about reproducing Figure 3, which is now Figure 6. And Figure 3, which is quite interesting, is not reproduced. Since extracting the data from the original plot was not trivial (involved extracting command in the PDF), it would be valuable for this data to be shared in the replication.
  • And even better, the code for extracting the data from the PDF could be made available too, as it should be valuable for other ReScience efforts.
  • Figure 3C is a bit busy. I would advise plotting differences with spikes and UEs in two different figures (this could allow to plots differences in UEs for Figure 1 as well).
  • Figure 3 is missing all units; it should be on the figures or in the caption.

Ease of use

  • Saving figure with Plotting_figure.py to disk should be by default. I use the Agg renderer with matplotlib, which means—being a non-interactive renderer—that show() has no effect for me.
  • I get a superposition of Figure 1 and Figure 2 when saving to disk (a detail, I can live without it being solved.):
  • When I talked about a requirements.txt file, I talked about a file compatible with pip (cf. https://pip.pypa.io/en/stable/user_guide/#requirements-files), so that all dependencies could be installed with a pip install -r requirements.txt. This is easily done by replacing the current one:
- Elephant>=0.5.0 (https://github.com/NeuralEnsemble/elephant)
- neo>=0.4.0 (https://github.com/NeuralEnsemble/python-neo)
- quantities>=0.9.0
- numpy>=1.6.2
- matplotlib>=1.5.1

by

Elephant>=0.4.0
neo>=0.4.0
quantities>=0.9.0
numpy>=1.6.2
matplotlib>=1.5.1

By the way, I assume you meant Elephant>=0.4.0, not 0.5.0 which is not released yet (latest release is 0.4.1, README.md is affected as well) ?

  • Optional setup and run instructions for non-apt-get users:
cd code
pip install -r requirements.txt
python Plotting_Figures.py
  • The notebook is missing three sets of parenthesis to be Python 3 compatible (and seems slightly behind the .py version), for instance, in the last cell:
  File "<ipython-input-5-90fba530d061>", line 16
    print 'calculating UE ...'
                             ^
SyntaxError: Missing parentheses in call to 'print'

Conclusions

Beyond those remarks, I fully recommend accepting the contribution.

The authors have done a remarkable job of addressing my comments. I particularly appreciate the quantitative analysis provided by the new Figure 3. So much that I have a couple of remarks about it:

Figure 3 not reproduced

  • The Plotting_figure.py code and the notebook talk about reproducing Figure 3, which is now Figure 6. And Figure 3, which is quite interesting, is not reproduced. Since extracting the data from the original plot was not trivial (involved extracting command in the PDF), it would be valuable for this data to be shared in the replication.
  • And even better, the code for extracting the data from the PDF could be made available too, as it should be valuable for other ReScience efforts.
  • Figure 3C is a bit busy. I would advise plotting differences with spikes and UEs in two different figures (this could allow to plots differences in UEs for Figure 1 as well).
  • Figure 3 is missing all units; it should be on the figures or in the caption.

Ease of use

  • Saving figure with Plotting_figure.py to disk should be by default. I use the Agg renderer with matplotlib, which means—being a non-interactive renderer—that show() has no effect for me.
  • I get a superposition of Figure 1 and Figure 2 when saving to disk (a detail, I can live without it being solved.):
  • When I talked about a requirements.txt file, I talked about a file compatible with pip (cf. https://pip.pypa.io/en/stable/user_guide/#requirements-files), so that all dependencies could be installed with a pip install -r requirements.txt. This is easily done by replacing the current one:
- Elephant>=0.5.0 (https://github.com/NeuralEnsemble/elephant)
- neo>=0.4.0 (https://github.com/NeuralEnsemble/python-neo)
- quantities>=0.9.0
- numpy>=1.6.2
- matplotlib>=1.5.1

by

Elephant>=0.4.0
neo>=0.4.0
quantities>=0.9.0
numpy>=1.6.2
matplotlib>=1.5.1

By the way, I assume you meant Elephant>=0.4.0, not 0.5.0 which is not released yet (latest release is 0.4.1, README.md is affected as well) ?

  • Optional setup and run instructions for non-apt-get users:
cd code
pip install -r requirements.txt
python Plotting_Figures.py
  • The notebook is missing three sets of parenthesis to be Python 3 compatible (and seems slightly behind the .py version), for instance, in the last cell:
  File "<ipython-input-5-90fba530d061>", line 16
    print 'calculating UE ...'
                             ^
SyntaxError: Missing parentheses in call to 'print'

Conclusions

Beyond those remarks, I fully recommend accepting the contribution.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 19, 2017

Member

@benureau Thanks for the review !
@Vahidrostami Congratulations, your submission is accepted. Could you address the remaining minor comments before we proceed with publication ?

Member

rougier commented Apr 19, 2017

@benureau Thanks for the review !
@Vahidrostami Congratulations, your submission is accepted. Could you address the remaining minor comments before we proceed with publication ?

@rougier rougier referenced this pull request in ReScience/ReScience Apr 19, 2017

Open

Article number request #48

@khinsen

This comment has been minimized.

Show comment
Hide comment
@khinsen

khinsen Apr 20, 2017

Following a recent discussion (ReScience/ReScience#43) about the non-reproducibility of articles published in ReScience, I did a quick test on this one using the accepted but not yet published code. As explained in the earlier discussion, all I checked is if I can install the dependencies and run the software - I don't look at the results, since I am not competent in judging them anyway.

First remark: the instructions quote version numbers for everything except Python itself. In particular, it isn't clear if I should use Python 2 or Python 3.

Using Python 3.5 under macOS 10.11, I installed the dependencies via pip and ran Plotting_Figures.py. The output:

plotting Figure 1 with trigger:  PS_4 ...
plotting raster plot ...
plotting Spike Rates ...
plotting Raw Coincidences ...
plotting emp. and exp. coincidences rate ...
plotting Surprise ...
plotting UEs ...

plotting Figure 2 with trigger:  RS_4 ...
plotting raster plot ...
plotting Spike Rates ...
plotting Raw Coincidences ...
plotting emp. and exp. coincidences rate ...
plotting Surprise ...
plotting UEs ...

plotting Figure 3 ...
calculating UE ...
calculating UE ...
/Users/hinsen/Temp/ReScience-submission-VRostami-JIto-MDenker-SGruen-2016/code/utils.py:455: RuntimeWarning: invalid value encountered in greater_equal
  Js_dict_lst[N_cnt]['Js'] >= Js_sig)[0]

If that warning is expected and harmless, it would be nice to mention this in the instructions.

khinsen commented Apr 20, 2017

Following a recent discussion (ReScience/ReScience#43) about the non-reproducibility of articles published in ReScience, I did a quick test on this one using the accepted but not yet published code. As explained in the earlier discussion, all I checked is if I can install the dependencies and run the software - I don't look at the results, since I am not competent in judging them anyway.

First remark: the instructions quote version numbers for everything except Python itself. In particular, it isn't clear if I should use Python 2 or Python 3.

Using Python 3.5 under macOS 10.11, I installed the dependencies via pip and ran Plotting_Figures.py. The output:

plotting Figure 1 with trigger:  PS_4 ...
plotting raster plot ...
plotting Spike Rates ...
plotting Raw Coincidences ...
plotting emp. and exp. coincidences rate ...
plotting Surprise ...
plotting UEs ...

plotting Figure 2 with trigger:  RS_4 ...
plotting raster plot ...
plotting Spike Rates ...
plotting Raw Coincidences ...
plotting emp. and exp. coincidences rate ...
plotting Surprise ...
plotting UEs ...

plotting Figure 3 ...
calculating UE ...
calculating UE ...
/Users/hinsen/Temp/ReScience-submission-VRostami-JIto-MDenker-SGruen-2016/code/utils.py:455: RuntimeWarning: invalid value encountered in greater_equal
  Js_dict_lst[N_cnt]['Js'] >= Js_sig)[0]

If that warning is expected and harmless, it would be nice to mention this in the instructions.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 20, 2017

Member

Thanks @khinsen, I totally forgot this discussion and your proposal of including a third reviewer not from the field to test for the correctness of the instruction.

@Vahidrostami Do you know if the warning is expected ?

Member

rougier commented Apr 20, 2017

Thanks @khinsen, I totally forgot this discussion and your proposal of including a third reviewer not from the field to test for the correctness of the instruction.

@Vahidrostami Do you know if the warning is expected ?

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Apr 21, 2017

@rougier This is expected. This happens because the array contains nan when it tests for greater_equal. This warning occurs with python 3 not python 2. As we wrote the whole code in python 2 this warning slipped our attention. However as the reviewer suggested we are making the code compatible with Python 3.

@khinsen Thanks for your test. In the final version we will take care of this warning.

@rougier This is expected. This happens because the array contains nan when it tests for greater_equal. This warning occurs with python 3 not python 2. As we wrote the whole code in python 2 this warning slipped our attention. However as the reviewer suggested we are making the code compatible with Python 3.

@khinsen Thanks for your test. In the final version we will take care of this warning.

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami Apr 23, 2017

@rougier
@benureau Thanks a lot for your comments. Please find below a detailed response to your comments:
"

Figure 3 not reproduced

  • The Plotting_figure.py code and the notebook talk about reproducing Figure 3, which is now Figure 6. And Figure 3, which is quite interesting, is not reproduced. Since extracting the data from the original plot was not trivial (involved extracting command in the PDF), it would be valuable for this data to be shared in the replication.
  • And even better, the code for extracting the data from the PDF could be made available too, as it should be valuable for other ReScience efforts.
    "

Regarding these two points, we are happy to share our code for extracting plot points from a PDF file, but we do not think that it should be included into the publication, for the following reason.
We implemented our code so that it does NOT directly read the original PDF file, but it reads an EPS file (which is a text file, much easier for a Python script to read) containing only the figure part of the original PDF.
We generated this EPS file, using Adobe Acrobat XI Pro on Windows OS, by deleting everything in the original PDF document except for the figure part and then exporting the result to an EPS file.
Thus, if we are to submit the code in a form testable by reviewers, we also need to submit this EPS file, which is a modified version of the original publication, but we guess that re-distributing a modified version of a publication is generally prohibited in most licenses.
We would suggest, instead, to publish the code only (but with a documentation on how to prepare the EPS file) as supplementary information or something like.
@rougier We wonder if such option is available in ReScience journal.

"

  • Figure 3C is a bit busy. I would advise plotting differences with spikes and UEs in two different figures (this could allow to plots differences in UEs for Figure 1 as well).
    "
    As the reviewer suggested, we have separated the distribution of differences for spikes and UEs in two different subplots.
    Plotting the distribution of differences for UEs in Figure 1 is not possible as the number of UEs extracted from the original publication differs from the reproduced one in Figure 1.

"

  • Figure 3 is missing all units; it should be on the figures or in the caption.
    "
    We thank the reviewer for pointing this out. We added the missing units to Figure 3.

"

Ease of use

"
We have corrected all the mentioned points.

@rougier
@benureau Thanks a lot for your comments. Please find below a detailed response to your comments:
"

Figure 3 not reproduced

  • The Plotting_figure.py code and the notebook talk about reproducing Figure 3, which is now Figure 6. And Figure 3, which is quite interesting, is not reproduced. Since extracting the data from the original plot was not trivial (involved extracting command in the PDF), it would be valuable for this data to be shared in the replication.
  • And even better, the code for extracting the data from the PDF could be made available too, as it should be valuable for other ReScience efforts.
    "

Regarding these two points, we are happy to share our code for extracting plot points from a PDF file, but we do not think that it should be included into the publication, for the following reason.
We implemented our code so that it does NOT directly read the original PDF file, but it reads an EPS file (which is a text file, much easier for a Python script to read) containing only the figure part of the original PDF.
We generated this EPS file, using Adobe Acrobat XI Pro on Windows OS, by deleting everything in the original PDF document except for the figure part and then exporting the result to an EPS file.
Thus, if we are to submit the code in a form testable by reviewers, we also need to submit this EPS file, which is a modified version of the original publication, but we guess that re-distributing a modified version of a publication is generally prohibited in most licenses.
We would suggest, instead, to publish the code only (but with a documentation on how to prepare the EPS file) as supplementary information or something like.
@rougier We wonder if such option is available in ReScience journal.

"

  • Figure 3C is a bit busy. I would advise plotting differences with spikes and UEs in two different figures (this could allow to plots differences in UEs for Figure 1 as well).
    "
    As the reviewer suggested, we have separated the distribution of differences for spikes and UEs in two different subplots.
    Plotting the distribution of differences for UEs in Figure 1 is not possible as the number of UEs extracted from the original publication differs from the reproduced one in Figure 1.

"

  • Figure 3 is missing all units; it should be on the figures or in the caption.
    "
    We thank the reviewer for pointing this out. We added the missing units to Figure 3.

"

Ease of use

"
We have corrected all the mentioned points.

@benureau

This comment has been minimized.

Show comment
Hide comment
@benureau

benureau Apr 24, 2017

@Vahidrostami: I think sharing the code for extracting the data from the EPS and descriptive instructions on how the EPS was created, along with the data generated and the code of Figure 3, would be perfectly fine for me.

@Vahidrostami: I think sharing the code for extracting the data from the EPS and descriptive instructions on how the EPS was created, along with the data generated and the code of Figure 3, would be perfectly fine for me.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 24, 2017

Member

Maybe you can illustrate the technique on a "fake" PDF file. Fro extracting the figure part, you can also use inkscape to open the PDF directly and then remove pretty much everything.

Member

rougier commented Apr 24, 2017

Maybe you can illustrate the technique on a "fake" PDF file. Fro extracting the figure part, you can also use inkscape to open the PDF directly and then remove pretty much everything.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 24, 2017

Member

@Vahidrostami Byt the way, it it's too much trouble, I think we can publish without this specific part.

Member

rougier commented Apr 24, 2017

@Vahidrostami Byt the way, it it's too much trouble, I think we can publish without this specific part.

@benureau

This comment has been minimized.

Show comment
Hide comment
@benureau

benureau Apr 24, 2017

@rougier It clearly goes way beyond this particular review, and IANAL, and the risks are possibly not worth taking, but this may fall under Fair Use (http://fairuse.stanford.edu/overview/fair-use/what-is-fair-use/). Transformative, partial, with the intent to comment, and "the public reaps benefits from your review".

Also, we could ask.

@rougier It clearly goes way beyond this particular review, and IANAL, and the risks are possibly not worth taking, but this may fall under Fair Use (http://fairuse.stanford.edu/overview/fair-use/what-is-fair-use/). Transformative, partial, with the intent to comment, and "the public reaps benefits from your review".

Also, we could ask.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 24, 2017

Member

I'll wait for @Vahidrostami to answer if this too much work or not. Since we've already accepted the publication, it might not be fair to delay the publication too much because of this.

Member

rougier commented Apr 24, 2017

I'll wait for @Vahidrostami to answer if this too much work or not. Since we've already accepted the publication, it might not be fair to delay the publication too much because of this.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier May 2, 2017

Member

@Vahidrostami What is your decision concerning @benureau request ? As I said, we can publish now without the requested supplementary code or we can wait a bit more if you intend to supply it. Juste tell us.

Member

rougier commented May 2, 2017

@Vahidrostami What is your decision concerning @benureau request ? As I said, we can publish now without the requested supplementary code or we can wait a bit more if you intend to supply it. Juste tell us.

@Vahidrostami

This comment has been minimized.

Show comment
Hide comment
@Vahidrostami

Vahidrostami May 2, 2017

@rougier As @benureau suggested, we have added now the following codes:

  • A script for extracting the data from the EPS file (extract_values_from_eps.py in the code folder). As we discussed before this code can not be run, as the reduced EPS file from Riehle et al (1997) does not exist. However, there is a detail description in the documentation of the code on how to generate the reduced EPS file.
  • The code for plotting Figure 3. This plot needs the extracted values from the EPS file which are provided with extracted_data.npy in the data folder.

@rougier As @benureau suggested, we have added now the following codes:

  • A script for extracting the data from the EPS file (extract_values_from_eps.py in the code folder). As we discussed before this code can not be run, as the reduced EPS file from Riehle et al (1997) does not exist. However, there is a detail description in the documentation of the code on how to generate the reduced EPS file.
  • The code for plotting Figure 3. This plot needs the extracted values from the EPS file which are provided with extracted_data.npy in the data folder.
@benureau

This comment has been minimized.

Show comment
Hide comment
@benureau

benureau May 2, 2017

@Vahidrostami: Thanks for this additional material.
@rougier: Everything is good for me.

benureau commented May 2, 2017

@Vahidrostami: Thanks for this additional material.
@rougier: Everything is good for me.

@ReScience ReScience locked and limited conversation to collaborators May 2, 2017

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier May 9, 2017

Member

Sorry, I'm late for publishing this submission, will do by end of next week.

Member

rougier commented May 9, 2017

Sorry, I'm late for publishing this submission, will do by end of next week.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier May 29, 2017

Member

This submission has been published and will soon appear at http://rescience.github.io/read/

DOI

Member

rougier commented May 29, 2017

This submission has been published and will soon appear at http://rescience.github.io/read/

DOI

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.