Review Request: R. Shimoura et. al.

[shimoura]

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: The cell-type specific cortical microcircuit: relating structure and activity in a full-scale spiking network model
Author(s): T.C. Potjans, M. Diesmann
Journal (or Conference): Cerebral Cortex
Year: 2014
DOI: 10.1093/cercor/bhs358
PDF: https://academic.oup.com/cercor/article/24/3/785/398560

Replication

Author(s): Renan O. Shimoura, Nilton L. Kamij1, Rodrigo F.O. Pena, Vinicius L. Cordeiro, Cesar C. Ceballos, Cecilia Romaro, and Antonio C. Roque
Repository: https://github.com/shimoura/ReScience-submission/tree/ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017
PDF: https://github.com/shimoura/ReScience-submission/blob/ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017/article/ShimouraR-KamijiNL-PenaRFO-CordeiroVL-CeballosCC-RomaroC-RoqueAC-2017.pdf
Keywords:
Language: Python
Domain: Computational Neuroscience

Results

• Article has been fully replicated
• Article has been partially replicated
• Article has not been replicated

Potential reviewers

Potential reviewer: @mstimberg

---

EDITOR

• Editor acknowledgment (@benoit-girard Nov 13, 2017)
• Reviewer 1 (@appukuttan-shailesh Nov 13, 2017)
• Reviewer 2 (@mstimberg Nov 14, 2017)
• Review 1 decision [accept] Apr 6, 2018
• Review 2 decision [accept] Apr 6, 2018
• Editor decision [accept] Apr 9, 2018

[rougier]

@shimoura Thanks for your submission. An editor will be soon assigned.

By the way, we have changed a little bit the template for submission so I may need to edit your text at the top (of the PR) to give all information. If it's ok for you, you don't need to do anything.

[rougier]

@benoit-girard Can you edit this submission ? (there are some missing information at the top of the PR but I will fix it today)

[shimoura]

It's ok for me. Thanks

[benoit-girard]

Yes, I will edit it.

[benoit-girard]

@appukuttan-shailesh and @mstimberg would you be interested in reviewing this submission?

[appukuttan-shailesh]

@benoit-girard : I would be interested in reviewing this.

[mstimberg]

@benoit-girard: yes, I'd be happy to review this submission, I can't work on it before next week, though.

[benoit-girard]

@mstimberg confirmed to me by email that he will review this submission, but will only be able to start next week.

edit: Well, he also confirmed above, I missed that...

[appukuttan-shailesh]

@mstimberg : find below Fig. 6 that I was able to produce by running the original protocol for 10 s in serial (in case you wish to use the same to draw any conclusions)

(more soon)

[appukuttan-shailesh]

Finally getting back to this review. I ran the simulations in serial model (thanks to @mstimberg for alerting me that the default value of serial wasn't as expected). Protocol #3 (for producing Fig. 6) is the longest and took well over two weeks! So (as a warning to future users) this should probably only be run on clusters or supercomputers, and not on local workstations.

Overall, I am happy with the replication of the figures using the Brian 2 variant of the original model. As @mstimberg mentioned, there are minor differences in some of the figures (such as Fig. 5C, 6, 7). Some of this, I believe, arises from the inherent stochasticity of the model and could be resolved by managing the seeds. The differences anyhow are minor, and do not affect the interpretation.

As previously conveyed, I am pleased to see the addition of the new section titled "Important information needed during replication" with info that is vital for a successful replication (highlighting omissions and mistakes in the original article). I too wasn't very satisfied with the final statement of the section titled "Propagation of Transient Thalamic Inputs"... "one can see the similarities" . This has now been updated, and though better than before, could probably still be reworded (I personally find the term 'think' not too confidence inspiring!). But this is a very minor remark, which the authors could take into consideration.

The authors opted not to attempt replication of one of the figures (Fig. 9 of the original article). Unless this contradicts ReScience policy, I am happy to accept the current submission as a "full replication" as in my view the major results have been replicated satisfactorily. The earlier discussion about full/partial replication got suspended without us arriving at a decision/conclusion, and probably needs more clarity going forward.

[mstimberg]

To give a (potentially final) summary of my review:
I am happy with the text and code, in my opinion this article can be accepted as it is for publication in ReScience. It replicates the results in the original publication well, and as @appukuttan-shailesh I would accept it as "full replication", but I don't have a strong opinion on this, either (does it actually get mentioned anywhere in the final article?) despite the missing Figure 9.
I was able to reproduce all figures except for Figure 6 with very minor differences and I can therefore confirm that the provided code produces the figures in the article. For Figure 6, the result provided by @appukuttan-shailesh also matches the figure provided by the authors enough to not make me doubt their results. However, if @benoit-girard thinks that it is necessary for me to replicate this result myself, then I will need a couple of additional weeks to get around doing that.

[A final remark not as a reviewer but as a Brian 2 developer: it might well be that the memory problems we encountered are due to a bug/regression in Brian: brian-team/brian2#933 ...]

[benoit-girard]

From what I read here, it seems that the paper is about to be accepted.
@rougier @khinsen : what do we do about this "full replication thing"? I am in favor of "full replication" (but it does not seem to be very important). @rougier suggested adding categories (but can we really define a full set of categories covering all the possible configurations?). Where do we go with regards to this? Do we stick to two? Do we label this paper "partial" or "full" replication?

[khinsen]

I vote for a pragmatic approach. Our categories aren't clear and haven't been applied consistently in the past. We should apply the rules valid at submission time for each paper, which means "not quite clear" for this one. Since the reviewers are happy to call it a full replication, so be it.

This doesn't mean we shouldn't clean up our categories, but that's a different issue (soon also in the GitHub sense of the word).

[mstimberg]

Just as a minor comment from my point of view about the "full replication/partial replication" issue: I think the question is not extremely pressing as this category does not appear anywhere in the PDF or the Zenodo archive, AFAICT. Of course, part of the appeal of ReScience is that the reviews are public as well, but I'd be certainly more cautious about calling the replication "full" if this was a big visible label on the paper.

[benoit-girard]

@mstimberg @appukuttan-shailesh
Dear reviewers, are you satisfied with the current state of the paper, or do you require additional changes (even minor) before acceptance?

[mstimberg]

@benoit-girard Yes, I am satisfied with the current state of the paper and recommend its acceptance (noting that I did not personally replicate the simulations for Figure 6, as explained in my earlier comments).

[appukuttan-shailesh]

@benoit-girard : Yes, I am happy with the (current) revised submission and recommend its acceptance.

[shimoura]

@mstimberg @appukuttan-shailesh Is it ok to delete the script fig6_sample.py? This script was used just to generate part of the figure 6 as commented on March 2, which is not part of the submitted article.

[mstimberg]

@shimoura : From my side, yes, I think it would be better to delete the script to avoid confusion.

[appukuttan-shailesh]

@shimoura : Yes please go ahead.

[benoit-girard]

OK, so the review process is over, both reviewer accepted the paper.
I will carry out the (relatively complex) publishing process in the coming days.

[rougier]

Yes, sorry for the complex publishing process. We need to write a (Python) script to try to handle it automatically.

[rougier]

@benoit-girard Any progress with the publication?

[rougier]

@benoit-girard 🛎

[benoit-girard]

I'm on it today...

[benoit-girard]

@shimoura sorry for the long publication delay. Can you provide me with a set of keywords?

[shimoura]

Dear @benoit, here are the keywords:
cerebral cortex; cortical microcircuit; large-scale models; layered network; specificity of connections;

[benoit-girard]

Thanks !

[benoit-girard]

EDITOR

This submission has been accepted for publication, and has been published at https://github.com/ReScience/ReScience/wiki/Current-Issue

[benoit-girard]

The version 1.1 of the paper has a correctly compiled bibliography. Its new DOI is: