Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Review Request: Maksimov Albada Diesmann #18
I request a review for the reproduction of the following paper:
A. Compte et al. “Cellular and network mechanisms of slow oscillatory activity (<1 Hz) and
I believe the original results have been faithfully reproduced as explained in the accompanying article.
The repository lives @ https://github.com/andruhamax/ReScience-submission/tree/maksimov-albada-diesmann
I'm trying to install and run the code on Ubuntu 16.04. First, a few remarks. I think it would help to:
I had a first go at running the code (under Ubuntu 14.04) and it worked with a few minor problems (see below) -- I did not yet look into the results or the code in any detail. I also agree with @rossant's comments (for the download link to NEST 2.8 maybe consider the Zenodo DOI?).
Not sure whether this is worth mentioning (as it is not specific to this code or even NEST), but I could not run the code within my anaconda environment due to a readline issue (see e.g. here: ContinuumIO/anaconda-issues#152). I used the workaround from that bug report (removing anaconda's readline and using
Installation readme.md file is updated to account for corrections from both reviewers.
Transition to Python 3 would require a major code revision, because differences between Python 2 and Python 3 are considerable. For a moment, we would like to stay with Python 2, which is still widely used.
My general assessment of the article, the code and its documentation is that it fulfills the quality standards of ReScience: all results are replicable using a standard tool of the community (NEST) when following the authors' instructions, and all code and analysis scripts are well documented. However, the replication of the original study remains qualitative, there are discrepancies between the results and the original results that need to be cleared up (see major issues below).
I cannot judge the details of the C++ implementation of the neuron/synapse models that are added to NEST itself, but 1) they seem to be based on established models; 2) they are clearly linked to the model description in the paper (e.g. the variable names are intuitive); 3) the code has a reasonable number of comments, stating important details (such as the expected scale of parameters).
Their approach of simplifying the synaptic model seems justified and they present simulations that support their claim that they successfully replicated the basic neuron and synapse dynamics. The authors also present an extensive investigation of the membrane input resistance measurements with some apparently relevant insights (I have to admit that I am not very familiar with these kind of measurements so I cannot judge this part on a detailed level).
The authors state (p.10): "However, despite close reproduction of the model constituents, the simulated network activity is characterized by unreasonably high firing rates due to the dominance of NMDA conductances far exceeding the potency of the opposing GABAergic inhibition. To achieve a network state characterized by comparable excitation and inhibition, we modify the synaptic strengths W of all synapse types". There are several issues here:
I wonder whether differences in the connectivity pattern could explain the different results. In the Compte et al. paper they state: "The neurons in the network are sparsely connected to each other through a fixed number of connections that are set at the beginning of the simulation. Neurons make 20±5 (SD) contacts to their postsynaptic partners (multiple contacts onto the same target, but no autapses, are allowed)." The authors of the present manuscript interpret this in a way that each excitatory cell connects (on average) to 20 other excitatory cells and to 20 other inhibitory cells, the same for inhibitory cells. I think there are other possible interpretations: e.g. each neuron could have 20 outgoing connections in total (of which 4/5 would go to excitatory cells since they outnumber excitatory cells by 4:1). Also, the authors of the present manuscript assume that all excitatory connections form one AMPA and one NMDA synapse (which means that excitatory synapses could be modeled as one AMPA-NMDA hybrid synapse) – the original paper does not make any clear statement on the distribution of AMPA vs. NMDA synapses, but in principle the above quote about the connectivity does not exclude that excitatory cells form AMPA and NMDA synapses independently.
The authors have to make their decisions and their understanding of the original paper clear and motivate why they chose the specific values they chose (and the conclusion should not deemphasize these changes as "minor modifications").
The results that the authors achieve with the adjusted weight values reproduce several of the results of the original paper (spontaneous up-down oscillations, dynamics of membrane resistance and ionic conductances) but also look different in quantitative terms: firing rate in Compte et al. is up to ~10Hz (exc. neurons) and ~20Hz (inh. neurons) while it is up to ~20Hz (exc. neurons) and ~35Hz (inh. neurons) in the present manuscript. The dynamics of the firing rate during an up-state is also quite different, it is uni-modal in the Compte et al. study but has a strong peak followed by a lower level plateau in the present manuscript. Also, the wave-like propagation is far from clear and inhibitory neurons do not seem to fire before excitatory neurons as in the original study. Finally, the length of one Up period is ~2.5s, followed by a Down period of approximately the same length whereas in the original paper an Up periods lasts ~ 0.5s to 1s, followed by a Down period of ~2.5s.
The authors chose to focus on a subset of the results of the original paper (which is of course fine in general), it's a pity that the paper does not include some robustness investigation, though. In particular, some manipulations like stronger or weaker inhibition/excitation (Fig. 6B,C in the Compte et al. paper) could be performed with the present code just by changing one parameter value.
With the changes made by the authors I am now happy with the installation instructions and I can install NEST and their code following them step by step. The only thing I am not sure about is whether it is necessary to mention the installation from the deadsnake PPA. This is not necessary for current versions of Ubuntu and while it may become necessary in the future, relying on an unofficial PPA is not great. I would maybe link to it with something like "if your Ubuntu distribution no longer provides a Python 2.7 package, you can use the following PPA". In any case, telling users how to install Python 2 on their machine (which they most likely already have) without telling them how to install numpy and matplotlib (which they might not have), is a bit odd.
The code is clear and well-documented. A few minor remarks:
I didn't have time to look at the article yet. I'll do it when I come back from vacation in 10 days.
This submission's quality is good, the code and article are clear, and the results are qualitatively similar to the original article. Like @mstimberg, the main concern I have is that the network activity is not reproduced with the same synapse parameters, and the authors had to use different values than in the original implementation. I think this deserves further investigations and explanations.
Answer to reviewer 1:
Our analysis suggests that the activation of single synapses elicits different postsynaptic responses in the authors' work and in our non-simplified implementation of the original synaptic dynamics. However, due to the absence of the source code, we cannot find the origin of this discrepancy.
In the revised manuscript, we explain this issue as follows:
“This difference from the original results might be a consequence of the different implementation of synaptic dynamics. Closer examination of the original Figure 5C suggests that unitary excitatory and inhibitory postsynaptic responses are approximately 0.5 nS during periods of network silence. This contradicts our implementation of the non-modified synaptic dynamics (see Fig. 2), where single synaptic activations (with synaptic weights as in the original model) would result in 5.4 and 0.8 nS postsynaptic response for AMPA and GABA channels, respectively. This suggests an inconsistency between the reported dynamics and that which was implemented already in the original work. Furthermore, we assumed the units of the function defining the dependence of the synaptic conductances on the presynaptic
“Besides the reasons mentioned above, differences between the simplified and original synaptic kinetics during network activity, not brought out by the simple inputs in Fig. 2, may contribute to the need for weight changes to obtain a network state similar to that in the original study. However, this is unlikely to account for the need to strongly scale synaptic weights relative to each other. In the absence of the original code and since our network implementation does not allow using the original synapse dynamics, we cannot resolve this issue here.”
“How were new parameter values chosen?”
We find that it is not possible to reach the desired network regime with a simple scaling of all synaptic weights, so that ratios stay unchanged. This is probably connected to discrepancies in the implementation of the synaptic dynamics, mentioned above. Therefore, we scaled synaptic weight values individually based on the ability of the combined set of weights to provide the desired network dynamics. This approach is in full agreement with the approach chosen in the original work, where the proper network dynamics is the key factor, rather than a close match of the synaptic weights to known biology.
In the revised manuscript, we clarify the basis for our choice of parameter values:
“To achieve an appropriate network regime, we modify the synaptic strengths W of all synapse types. Specifically, we match the firing rates, the duration of Up and Down states, the wave propagation speed, and the response to network stimulation.”
“The authors have to make their decisions and their understanding of the original paper clear and motivate why they chose the specific values they chose”
In our hands, multiple sets of synaptic weights could provide the desired network dynamics. We chose a particular realization from among these sets based on relative proximity to the originally reported weight values. We mention this in the revised manuscript:
“However, a range of different synaptic weight settings was able to produce qualitatively identical network activity. Therefore, we chose a set of weights from among these possibilities based on relative proximity to the originally reported weight values.”
“I wonder whether differences in the connectivity pattern could explain the different results.”
We completely agree with the reviewer in his interpretation of the network connectivity, and corrected our model accordingly. However, this change did not improve the network dynamics. Further adjustment of the synaptic weights was required.
“the authors of the present manuscript assume that all excitatory connections form one AMPA and one NMDA synapse”
From biology we know that cortical connections between excitatory neurons have both AMPA and NMDA receptors combined at postsynaptic sites. In the original paper, the authors speak about excitatory and inhibitory connections with no indication about separation into AMPA and NMDA types. Therefore, we think it is safe to assume that AMPA and NMDA are colocalized also in the original model.
“The results that the authors achieve...look different in quantitative terms”
For the revised version of our manuscript, we adjusted the parameters for a closer quantitative matching of the network dynamics. This includes firing rates, Up and Down state durations, wave propagation speed, and response to network stimulation (an additional figure (Fig.5) appeared). The duration of Up states in our model is approx. 0.6 sec, similar to their Figure 3. The duration of Down states in our model is approx. 2.6 sec, similar to their Figure 3.
“it's a pity that the paper does not include some robustness investigation”
To address this issue, in the revised version we analyze the network response to 10% modification of synaptic weights, as well as to blockade of AMPA, NDMA, and GABA channels (see top of p. 17 and new Figure 10).
The conclusion section is slightly rewritten to incorporate the minor changes due to the modification of the synaptic parameters.
“We provide a closer look at the method which the authors of the original study use to access neuronal conductance (or resistance). Their measure reflects an “instantaneous chord conductance”, which results in the model in values around 15 nS in isolated pyramidal neuron, and around 20 nS throughout network activity. In experimental works, the typical measure approximates a “steady-state slope conductance” (see [@jack1983] for classification), which results in a membrane conductance of less than 1 nS for a slightly depolarized isolated pyramidal neuron. Throughout network activity, this yields around 2 nS during Down states, strongly fluctuating membrane conductance during subthreshold Up state periods, and a strong negative conductance lasting on average 32 ms before each spike event. These low and even negative long-lasting conductances are typically not reported in the experimental literature. Therefore, the dynamics of membrane conductance in the model merits further investigation. The present model successfully reproduces the network response to blockade of AMPA, NMDA, or GABA channels. However, the sensitivity of the present model to changes in the synaptic strengths is weaker than in the original model.
We could not obtain the desired network regime with the synaptic weights reported in the original work after calibration to account for the change in the synaptic model. We suggest the synaptic dynamics to be the source of discrepancy, as suggested by the inconsistency between our synapse implementation and the results shown in the original Figure 5C.”
We corrected all the minor issues. Also, we noticed some superfluous variables in the neuron model code, and removed these. Further, the axial conductance now is called 'g_conn' for consistency with other NEST models.
The latest updates address all of my concerns. The discrepancies in synaptic weights to the original paper are still somewhat unsatisfying, but with the additional remarks in the text the observed differences and the authors' reasoning are now clear enough IMO. I therefore recommend the acceptance of the submission.
Just a few final minor remarks/suggestions (no need to do another round of reviews for those, I think):