New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review Request: R. Shimoura et. al. #43

Closed
wants to merge 46 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@shimoura

shimoura commented Nov 9, 2017

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: The cell-type specific cortical microcircuit: relating structure and activity in a full-scale spiking network model
Author(s): T.C. Potjans, M. Diesmann
Journal (or Conference): Cerebral Cortex
Year: 2014
DOI: 10.1093/cercor/bhs358
PDF: https://academic.oup.com/cercor/article/24/3/785/398560

Replication

Author(s): Renan O. Shimoura, Nilton L. Kamij1, Rodrigo F.O. Pena, Vinicius L. Cordeiro, Cesar C. Ceballos, Cecilia Romaro, and Antonio C. Roque
Repository: https://github.com/shimoura/ReScience-submission/tree/ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017
PDF: https://github.com/shimoura/ReScience-submission/blob/ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017/article/ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017.pdf
Keywords:
Language: Python
Domain: Computational Neuroscience

Results

  • Article has been fully replicated
  • Article has been partially replicated
  • Article has not been replicated

Potential reviewers

Potential reviewer: @mstimberg


EDITOR

  • Editor acknowledgment (@benoit-girard Nov 13, 2017)
  • Reviewer 1 (@appukuttan-shailesh Nov 13, 2017)
  • Reviewer 2 (@mstimberg Nov 14, 2017)
  • Review 1 decision [accept] Apr 6, 2018
  • Review 2 decision [accept] Apr 6, 2018
  • Editor decision [accept] Apr 9, 2018

shimoura added some commits Nov 8, 2017

Add files via upload
Initial commit
Add files via upload
Initial commit
Add files via upload
Initial commit
Rename ShimouraR-KamijiNL-PenaRFO-CordeiroVL-RomaroC-CeballosCC-Roque…
…AC-2017.bib to ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017.bib
Rename ShimouraR-KamijiNL-PenaRFO-CordeiroVL-RomaroC-CeballosCC-Roque…
…AC-2017.tex to ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017.tex

@mstimberg mstimberg referenced this pull request Nov 9, 2017

Closed

Review Request: R. Shimoura et. al. #42

0 of 6 tasks complete
@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 10, 2017

Member

@shimoura Thanks for your submission. An editor will be soon assigned.

By the way, we have changed a little bit the template for submission so I may need to edit your text at the top (of the PR) to give all information. If it's ok for you, you don't need to do anything.

Member

rougier commented Nov 10, 2017

@shimoura Thanks for your submission. An editor will be soon assigned.

By the way, we have changed a little bit the template for submission so I may need to edit your text at the top (of the PR) to give all information. If it's ok for you, you don't need to do anything.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Nov 10, 2017

Member

@benoit-girard Can you edit this submission ? (there are some missing information at the top of the PR but I will fix it today)

Member

rougier commented Nov 10, 2017

@benoit-girard Can you edit this submission ? (there are some missing information at the top of the PR but I will fix it today)

@shimoura

This comment has been minimized.

Show comment
Hide comment
@shimoura

shimoura Nov 10, 2017

It's ok for me. Thanks

shimoura commented Nov 10, 2017

It's ok for me. Thanks

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard Nov 13, 2017

Yes, I will edit it.

benoit-girard commented Nov 13, 2017

Yes, I will edit it.

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard Nov 13, 2017

@appukuttan-shailesh and @mstimberg would you be interested in reviewing this submission?

benoit-girard commented Nov 13, 2017

@appukuttan-shailesh and @mstimberg would you be interested in reviewing this submission?

@appukuttan-shailesh

This comment has been minimized.

Show comment
Hide comment
@appukuttan-shailesh

appukuttan-shailesh Nov 13, 2017

@benoit-girard : I would be interested in reviewing this.

appukuttan-shailesh commented Nov 13, 2017

@benoit-girard : I would be interested in reviewing this.

@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Nov 14, 2017

@benoit-girard: yes, I'd be happy to review this submission, I can't work on it before next week, though.

mstimberg commented Nov 14, 2017

@benoit-girard: yes, I'd be happy to review this submission, I can't work on it before next week, though.

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard Nov 14, 2017

@mstimberg confirmed to me by email that he will review this submission, but will only be able to start next week.

edit: Well, he also confirmed above, I missed that...

benoit-girard commented Nov 14, 2017

@mstimberg confirmed to me by email that he will review this submission, but will only be able to start next week.

edit: Well, he also confirmed above, I missed that...

@appukuttan-shailesh

This comment has been minimized.

Show comment
Hide comment
@appukuttan-shailesh

appukuttan-shailesh Mar 27, 2018

@mstimberg : find below Fig. 6 that I was able to produce by running the original protocol for 10 s in serial (in case you wish to use the same to draw any conclusions)
image

(more soon)

appukuttan-shailesh commented Mar 27, 2018

@mstimberg : find below Fig. 6 that I was able to produce by running the original protocol for 10 s in serial (in case you wish to use the same to draw any conclusions)
image

(more soon)

@appukuttan-shailesh

This comment has been minimized.

Show comment
Hide comment
@appukuttan-shailesh

appukuttan-shailesh Mar 28, 2018

Finally getting back to this review. I ran the simulations in serial model (thanks to @mstimberg for alerting me that the default value of serial wasn't as expected). Protocol #3 (for producing Fig. 6) is the longest and took well over two weeks! So (as a warning to future users) this should probably only be run on clusters or supercomputers, and not on local workstations.

Overall, I am happy with the replication of the figures using the Brian 2 variant of the original model. As @mstimberg mentioned, there are minor differences in some of the figures (such as Fig. 5C, 6, 7). Some of this, I believe, arises from the inherent stochasticity of the model and could be resolved by managing the seeds. The differences anyhow are minor, and do not affect the interpretation.

As previously conveyed, I am pleased to see the addition of the new section titled "Important information needed during replication" with info that is vital for a successful replication (highlighting omissions and mistakes in the original article). I too wasn't very satisfied with the final statement of the section titled "Propagation of Transient Thalamic Inputs"... "one can see the similarities" . This has now been updated, and though better than before, could probably still be reworded (I personally find the term 'think' not too confidence inspiring!). But this is a very minor remark, which the authors could take into consideration.

The authors opted not to attempt replication of one of the figures (Fig. 9 of the original article). Unless this contradicts ReScience policy, I am happy to accept the current submission as a "full replication" as in my view the major results have been replicated satisfactorily. The earlier discussion about full/partial replication got suspended without us arriving at a decision/conclusion, and probably needs more clarity going forward.

appukuttan-shailesh commented Mar 28, 2018

Finally getting back to this review. I ran the simulations in serial model (thanks to @mstimberg for alerting me that the default value of serial wasn't as expected). Protocol #3 (for producing Fig. 6) is the longest and took well over two weeks! So (as a warning to future users) this should probably only be run on clusters or supercomputers, and not on local workstations.

Overall, I am happy with the replication of the figures using the Brian 2 variant of the original model. As @mstimberg mentioned, there are minor differences in some of the figures (such as Fig. 5C, 6, 7). Some of this, I believe, arises from the inherent stochasticity of the model and could be resolved by managing the seeds. The differences anyhow are minor, and do not affect the interpretation.

As previously conveyed, I am pleased to see the addition of the new section titled "Important information needed during replication" with info that is vital for a successful replication (highlighting omissions and mistakes in the original article). I too wasn't very satisfied with the final statement of the section titled "Propagation of Transient Thalamic Inputs"... "one can see the similarities" . This has now been updated, and though better than before, could probably still be reworded (I personally find the term 'think' not too confidence inspiring!). But this is a very minor remark, which the authors could take into consideration.

The authors opted not to attempt replication of one of the figures (Fig. 9 of the original article). Unless this contradicts ReScience policy, I am happy to accept the current submission as a "full replication" as in my view the major results have been replicated satisfactorily. The earlier discussion about full/partial replication got suspended without us arriving at a decision/conclusion, and probably needs more clarity going forward.

@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Mar 29, 2018

To give a (potentially final) summary of my review:
I am happy with the text and code, in my opinion this article can be accepted as it is for publication in ReScience. It replicates the results in the original publication well, and as @appukuttan-shailesh I would accept it as "full replication", but I don't have a strong opinion on this, either (does it actually get mentioned anywhere in the final article?) despite the missing Figure 9.
I was able to reproduce all figures except for Figure 6 with very minor differences and I can therefore confirm that the provided code produces the figures in the article. For Figure 6, the result provided by @appukuttan-shailesh also matches the figure provided by the authors enough to not make me doubt their results. However, if @benoit-girard thinks that it is necessary for me to replicate this result myself, then I will need a couple of additional weeks to get around doing that.

[A final remark not as a reviewer but as a Brian 2 developer: it might well be that the memory problems we encountered are due to a bug/regression in Brian: brian-team/brian2#933 ...]

mstimberg commented Mar 29, 2018

To give a (potentially final) summary of my review:
I am happy with the text and code, in my opinion this article can be accepted as it is for publication in ReScience. It replicates the results in the original publication well, and as @appukuttan-shailesh I would accept it as "full replication", but I don't have a strong opinion on this, either (does it actually get mentioned anywhere in the final article?) despite the missing Figure 9.
I was able to reproduce all figures except for Figure 6 with very minor differences and I can therefore confirm that the provided code produces the figures in the article. For Figure 6, the result provided by @appukuttan-shailesh also matches the figure provided by the authors enough to not make me doubt their results. However, if @benoit-girard thinks that it is necessary for me to replicate this result myself, then I will need a couple of additional weeks to get around doing that.

[A final remark not as a reviewer but as a Brian 2 developer: it might well be that the memory problems we encountered are due to a bug/regression in Brian: brian-team/brian2#933 ...]

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard Apr 4, 2018

From what I read here, it seems that the paper is about to be accepted.
@rougier @khinsen : what do we do about this "full replication thing"? I am in favor of "full replication" (but it does not seem to be very important). @rougier suggested adding categories (but can we really define a full set of categories covering all the possible configurations?). Where do we go with regards to this? Do we stick to two? Do we label this paper "partial" or "full" replication?

benoit-girard commented Apr 4, 2018

From what I read here, it seems that the paper is about to be accepted.
@rougier @khinsen : what do we do about this "full replication thing"? I am in favor of "full replication" (but it does not seem to be very important). @rougier suggested adding categories (but can we really define a full set of categories covering all the possible configurations?). Where do we go with regards to this? Do we stick to two? Do we label this paper "partial" or "full" replication?

@khinsen

This comment has been minimized.

Show comment
Hide comment
@khinsen

khinsen Apr 4, 2018

I vote for a pragmatic approach. Our categories aren't clear and haven't been applied consistently in the past. We should apply the rules valid at submission time for each paper, which means "not quite clear" for this one. Since the reviewers are happy to call it a full replication, so be it.

This doesn't mean we shouldn't clean up our categories, but that's a different issue (soon also in the GitHub sense of the word).

khinsen commented Apr 4, 2018

I vote for a pragmatic approach. Our categories aren't clear and haven't been applied consistently in the past. We should apply the rules valid at submission time for each paper, which means "not quite clear" for this one. Since the reviewers are happy to call it a full replication, so be it.

This doesn't mean we shouldn't clean up our categories, but that's a different issue (soon also in the GitHub sense of the word).

@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Apr 4, 2018

Just as a minor comment from my point of view about the "full replication/partial replication" issue: I think the question is not extremely pressing as this category does not appear anywhere in the PDF or the Zenodo archive, AFAICT. Of course, part of the appeal of ReScience is that the reviews are public as well, but I'd be certainly more cautious about calling the replication "full" if this was a big visible label on the paper.

mstimberg commented Apr 4, 2018

Just as a minor comment from my point of view about the "full replication/partial replication" issue: I think the question is not extremely pressing as this category does not appear anywhere in the PDF or the Zenodo archive, AFAICT. Of course, part of the appeal of ReScience is that the reviews are public as well, but I'd be certainly more cautious about calling the replication "full" if this was a big visible label on the paper.

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard Apr 6, 2018

@mstimberg @appukuttan-shailesh
Dear reviewers, are you satisfied with the current state of the paper, or do you require additional changes (even minor) before acceptance?

benoit-girard commented Apr 6, 2018

@mstimberg @appukuttan-shailesh
Dear reviewers, are you satisfied with the current state of the paper, or do you require additional changes (even minor) before acceptance?

@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Apr 6, 2018

@benoit-girard Yes, I am satisfied with the current state of the paper and recommend its acceptance (noting that I did not personally replicate the simulations for Figure 6, as explained in my earlier comments).

mstimberg commented Apr 6, 2018

@benoit-girard Yes, I am satisfied with the current state of the paper and recommend its acceptance (noting that I did not personally replicate the simulations for Figure 6, as explained in my earlier comments).

@appukuttan-shailesh

This comment has been minimized.

Show comment
Hide comment
@appukuttan-shailesh

appukuttan-shailesh Apr 6, 2018

@benoit-girard : Yes, I am happy with the (current) revised submission and recommend its acceptance.

appukuttan-shailesh commented Apr 6, 2018

@benoit-girard : Yes, I am happy with the (current) revised submission and recommend its acceptance.

@shimoura

This comment has been minimized.

Show comment
Hide comment
@shimoura

shimoura Apr 6, 2018

@mstimberg @appukuttan-shailesh Is it ok to delete the script fig6_sample.py? This script was used just to generate part of the figure 6 as commented on March 2, which is not part of the submitted article.

shimoura commented Apr 6, 2018

@mstimberg @appukuttan-shailesh Is it ok to delete the script fig6_sample.py? This script was used just to generate part of the figure 6 as commented on March 2, which is not part of the submitted article.

@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Apr 6, 2018

@shimoura : From my side, yes, I think it would be better to delete the script to avoid confusion.

mstimberg commented Apr 6, 2018

@shimoura : From my side, yes, I think it would be better to delete the script to avoid confusion.

@appukuttan-shailesh

This comment has been minimized.

Show comment
Hide comment
@appukuttan-shailesh

appukuttan-shailesh Apr 6, 2018

@shimoura : Yes please go ahead.

appukuttan-shailesh commented Apr 6, 2018

@shimoura : Yes please go ahead.

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard Apr 9, 2018

OK, so the review process is over, both reviewer accepted the paper.
I will carry out the (relatively complex) publishing process in the coming days.

benoit-girard commented Apr 9, 2018

OK, so the review process is over, both reviewer accepted the paper.
I will carry out the (relatively complex) publishing process in the coming days.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 9, 2018

Member

Yes, sorry for the complex publishing process. We need to write a (Python) script to try to handle it automatically.

Member

rougier commented Apr 9, 2018

Yes, sorry for the complex publishing process. We need to write a (Python) script to try to handle it automatically.

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier

rougier Apr 23, 2018

Member

@benoit-girard Any progress with the publication?

Member

rougier commented Apr 23, 2018

@benoit-girard Any progress with the publication?

@rougier

This comment has been minimized.

Show comment
Hide comment
@rougier
Member

rougier commented May 2, 2018

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard May 3, 2018

I'm on it today...

benoit-girard commented May 3, 2018

I'm on it today...

@ReScience ReScience locked as resolved and limited conversation to collaborators May 3, 2018

@ReScience ReScience unlocked this conversation May 3, 2018

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard May 3, 2018

@shimoura sorry for the long publication delay. Can you provide me with a set of keywords?

benoit-girard commented May 3, 2018

@shimoura sorry for the long publication delay. Can you provide me with a set of keywords?

@shimoura

This comment has been minimized.

Show comment
Hide comment
@shimoura

shimoura May 3, 2018

Dear @benoit, here are the keywords:
cerebral cortex; cortical microcircuit; large-scale models; layered network; specificity of connections;

shimoura commented May 3, 2018

Dear @benoit, here are the keywords:
cerebral cortex; cortical microcircuit; large-scale models; layered network; specificity of connections;

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard commented May 7, 2018

Thanks !

@ReScience ReScience locked and limited conversation to collaborators May 7, 2018

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard May 7, 2018

EDITOR

This submission has been accepted for publication, and has been published at https://github.com/ReScience/ReScience/wiki/Current-Issue

DOI

benoit-girard commented May 7, 2018

EDITOR

This submission has been accepted for publication, and has been published at https://github.com/ReScience/ReScience/wiki/Current-Issue

DOI

@rougier rougier closed this May 9, 2018

@benoit-girard

This comment has been minimized.

Show comment
Hide comment
@benoit-girard

benoit-girard May 9, 2018

The version 1.1 of the paper has a correctly compiled bibliography. Its new DOI is:

DOI

benoit-girard commented May 9, 2018

The version 1.1 of the paper has a correctly compiled bibliography. Its new DOI is:

DOI

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.