Origin of the LIF parameter defaults

[LennardBo]

Hello everyone!
I've been using Norse extensively recently in my classification experiments. I'm trying to get to the bottom of the default LIF parameters and how they were chosen. Here, among many, many other locations in the code is where the default LIF parameters are instantiated. Now, v_leak and v_th are somewhat reasonably chosen at 0 and 1. In the literature I found that some people use ANNs to initialize those.

However, it eludes why tau_syn_inv and tau_mem_inv are chosen tas 200 and 100 respectively. I assume it relates to the default time step value of 0.001 as seen here but I still don't quite get it. Is it based on a paper? Or just some empiric results?
The same goes for the value for alpha (=100). I get it's related to the gradient of the heaviside function (as it's used here) but I don't see the origin of this particular default value.

I tried reading up on the Wikipedia links and the online book referenced here but they did not quite help me.

I hope I'm not the only one wondering about that and others will appreciate answers to my questions as well. Thanks a lot to anyone who can provide me with some hints about these questions.

[Jegp]

Thanks for the great, but tough, question.
It's great because the parameter values are obviously important and because we haven't done a good enough job to document where they come from. Most of our values are taken from existing neuron simulators. In the case of the LIF parameters, we've been heavily inspired by PyNN: https://github.com/NeuralEnsemble/PyNN/blob/d8056fa956998b031a1c3689a528473ed2bc0265/pyNN/standardmodels/cells.py#L83. I.e. tau_syn would be 5 ms and tau_syn_inv 1/5 ms (we use the inverse by default for efficiency; also, note the biologically plausible values for the LIF further below.
Regarding the alpha value of 100, that's based on a scaling factor for the SuperSpike surrogate gradient method that we set to better facilitate learning. Alpha inversely scales the impact of the forward signal:

norse/norse/torch/functional/superspike.py

Line 26 in 73d7c53

grad = grad_input / (alpha * torch.abs(inp) + 1.0).pow(

Your question is also tough to answer, because appealing to authority (PyNN) doesn't really absolve us from the responsibility of either 1) documenting why the values are the way they are or 2) pick better values. Similar values appear in other simulators, but I've never found a biological reason for this. I also haven't done extensive research on the matter. I'll put a few leads out to some contacts - would be great to know!
Finally, we have actually discussed the option to experimentally find better values for the parameter values, but never got around to it. It would require a somewhat systematic analysis of the domain/expected range of the neuron models, which might constrain them unnecessarily.

Was that helpful?

[LennardBo]

Hello Jens, thank you for taking the time to answer so thoroughly! And sorry that I answer this late, I've been on vacation in the meantime.

For me, it's sufficient to know that they're not based on nothing and that it might be worthwhile exploring them in a systematic manner. I could also imagine that depending on the specific use case, the optimal values might differ quite a bit.
Ihave to ask though: Your tau_syn corresponds to PyNN's but your tau_mem not quite (0.01 or 10ms in Norse vs 20ms in PyNN). Was there a specific reason for changing it or does it correspond to some other line in PyNN after all?

In the end, I'd suggest adding your reasoning for the default values to the documentation at the most important locations - most people will be satisfied with that and for proper scientific work, some explanation is needed when people do not want to change the defaults (which I guess occurs often).

Lastly, I also wanted to add that I'm really fond of your work and wanted to thank you for all your efforts!

[Jegp]

At the moment I don't have a better reply than "that's what other people have been doing". I can cite other sources that use similar time constants, but I would rather see if we can derive some form of stable intuition/proof behind the time constants. I actually reached out to other experts in the field, and one simply just replied they're using 10ms for the membrane constant because it's a neat number. Note that I don't blame them!

One thing I've done a few times is to train the time constants, just like one would train weight parameters. Which is handy, because 1) the exact value becomes less important and 2) the numbers will adapt to the problem. Especially when doing optimization, the numbers from PyNN/Nest/Neuron/etc. that are (likely) biologically inspired, won't have much merit.

Thanks for the shout-out! It's deeply appreciated :-) Consider giving us a mention on Twitter (#norseai) or advertise it to your friends.