Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
To spike or not to spike #1
Small disclaimer: I am yet another phd student whose main scope of research happened to be SNN. What I am to say below is only based on what I currently think I know. I might be very wrong, so please do not judge me too harshly. Also, when speaking science, I am quite raw, please do not take any offence. None is intended.
Spiking neurons can encompass a lot of different models.
On the computational neuroscience side, the problems are many-fold:
All that to say that the debate is still open, and that in an extended way, what you coded could ''maybe'' be considered as an SNN. And how exactly real neurons learn is still not totally clear, including the question of the substrate of the learning system (only the neurons? neurons + glial cells? neuron membranes? Chemical pathways? DNA -what if your learning rule is a conditional code on a small patch of DNA, triggered through chemical pathways which were resulting from the membrane voltage local activations? - implicitly, this is also the question of the scale at which the information is ''really'' processed and stored, and at which the ''intelligence'' is-).
Now, in the deep learning papers, and learning models of SNN, SNNs are more to refer to all-or-none bit transmissions between neurons.
About the SNN not being a RNN, again tricky depending on what you define to be a SNN. But the version you implemented, for me is most definitely a RNN. I could draw a NN graph with recurrent connections. A RNN is simply a NN with loops, it does not say anything about where the loop should be. Once you have a loop, you have a notion of a stored state (a memory system) which is kind of what the membrane voltage of simplified spiking neuron models also enables.
On the brain rhythms, you might give a look at Jun Tani's work on Multiple-timescale NN such as in: 'Emergence of Functional Hierarchy in a Multiple-Timescale Neural Network Model: A Humanoid Robot Experiment'. Also, though not yet there in my reading, you might have a look at neural oscillations in SNN (but these are usually the latest chapters of the spiking neuron books). Jun Tani's work is based in a big part on Predictive Coding -learning from the divergence between what the system's model of the world predicts and the real sensory pieces of information or their representations-, which is not the directly the same as auto-encoders plus CD. But is it that different? (True question, I am not good enough in auto-encoders and CD to be able to answer that yet.)
Last but not least, you can still check if your specific implementation and ideas have been put into a paper, and if they had not been, develop the mathematical justification for the BP-STDP link in that peculiar configuration of yours, or try experimentally your other ideas, and check if these had been published or not.
Haha, "yet another PhD student", hilarious. Thanks for the feedback!
You say "you would have to replace your relu by a heaviside": you are totally right. But the heaviside's derivative is 0 so I absolutely needed something else. Perhaps a heaviside with a tricked derivative of 1 could do the job?
For the RNN thing, yeah, it's debatable. I prefer not to call this an RNN as it's not literally connected to itself with another weight "w", although I see how you could say it's somehow like an RNN. I'd like to see the SNN as not a discrete system (e.g.: it could be implemented with a thread per neuron and the times at which firing happen would be the only events processed by the threads, and launching an update of each post-synaptic threads it's connected to. Those "events" could happen at any continuous time value. Although thinking like that is a bit pixelated, it'd be quite energy-efficient in computers.