Add Prediction to SP & Per-Cell Feedforward Dendrites

[fergalbyrne]

Jeff's theory reflects the fact that predictive cells are partially depolarised by distal inputs, and so will have enhanced activations when provided with feedforward inputs. This is missing in the SP. In addition, the SP uses a common dendrite for all the cells, whereas they have their own dendrites in the theory. This can be remedied with just a few lines of code and a small memory/CPU cost. The benefits can be established by swarming on this enhancement for different types of data.
More on this (with pseudocode) here: http://inbits.com/2013/11/adding-prediction-to-the-nupic-spatial-pooler/

[rhyolight]

@subutai / @scottpurdy / @chetan51 Any comments?

[scottpurdy]

Our current TM theory relies pretty heavily on the current columnar properties. Things like bursting or intra-columnar inhibition, which is necessary for maintaining a large capacity for representing high order sequences, rely on well defined minicolumns. But with this proposal you have no way of ensuring that cells in a column are connected to the same inputs unless you reduce the receptive field dramatically. But then you lose the ability to pool spatially.

@subutai - do you see things differently?

@fergalbyrne - With your change, which allows cells in a column to respond to different feedforward activations, do you see a way to retain the high order properties and capacity in the temporal memory?

[subutai]

@scottpurdy I agree with you generally. On the other hand I'm fine with experimentation and Fergal's point is also a good one. We could have some experimental versions as subclasses for example. I think it is up to @fergalbyrne or someone else to make a concrete pull request, ideally with some examples that demonstrate its efficacy.

[chessweb01]

How about combining the classes TemporalMemory (or TP?) and SpatialPooler in a way that realizes @fergalbyrne 's idea?

[cogmission]

In my opinion, ideally, there should be a separate structure that contains the column cell matrix and state. The Connections object is currently being separated out to fulfill this role. (I also think at that point (or after) the state should be separated out, and the structure should be moved into an entity entitled "Layer"). Layers have a columnar/cell matrix structure, and the state should exist in the "Connections" object separately.

After that the columns could conceivably be "primed" for or have some variable representing a depolarized state - since the reason for a single dendrite is to represent the cells in a micro-column having the same feed forward response. The "overlap" scoring could include a "boost" from the existence of depolarized cells in a certain column; arising from distal dendritic/synaptic activity.

If understand @fergalbyrne 's ideas correctly?

[david-ragazzi]

How about combining the classes TemporalMemory (or TP?) and SpatialPooler in a way that realizes @fergalbyrne 's idea?

I was quite favor of combining spatial and temporal classes a time ago but I changed my mind as I follow nupic progress and know better its code. As a class evoluate or even is replaced then all combined super classes also need be updated (look the old spatial pooler [AKA FDRCSpatial2] case, for example). Furthermore, we would need have several combined classes to attend each implementation (for example, Comb1=SP+TM_v1, Comb2=SP+TM_v2, etc). Actually Network API already does such combination. I'm favor of keep implementations in an independent way in order to avoid such additional maintenance work BUT make such implementations share the same abstract model of objects to store states of Columns, Cells, Segments, etc. This way we could change the spatial and temporal implementations but without necessarily change the structure of such neuronal model of objects. In other words, a neuronal model only would store states, not would perform spatial or temporal processes. In my opinion, Connections class is a first step in this path.

[david-ragazzi]

In my opinion, ideally, there should be a separate structure that contains the column cell matrix and state. The Connections object is currently being separated out to fulfill this role. (I also think at that point (or after) the state should be separated out, and the structure should be moved into an entity entitled "Layer"). Layers have a columnar/cell matrix structure, and the state should exist in the "Connections" object separately.

@cogmission oooohh.. We was thinking almost the same (i.e. separate process than state).. You was faster than me!! 👊

[fergalbyrne]

Thanks guys for hopping on to this, it's deeply appreciated. I'll be making some comments which might help us organise our thinking and get things rolling.

[fergalbyrne]

OK. We're going to add a repo to nupic-community for algorithm research, and do everything external to nupic itself (this is how nupic.research works, using nupic as a library and subclassing). This way we avoid the whole PR-CI cycle, and we can set up our own tests and prototype quickly. Any comments or objections? Also, not sure who is to set that up... @subutai might know..

[david-ragazzi]

@fergalbyrne Do you mean a front-end for nupic?

[fergalbyrne]

@cogmission and @david-ragazzi I think we should just do the simplest thing that'll work, which is to create a single PACLALayer class which has the interfaces of both SP and TM. If we need to, we can create a PASPLayer and a PATMLayer which inherit from SP and TM respectively, but just proxy to the same PACLALayer to do the work.

[fergalbyrne]

@david-ragazzi Not sure what that means. @subutai suggested this, he uses this model for both nupic.research and his own repo.

[fergalbyrne]

OK, I have to go to a meeting. I'll try and rejoin the conversation in a couple of hours. Thanks again!

[fergalbyrne]

(as he leaves) Thanks Matt for https://github.com/nupic-community/community.research !

[subutai]

@david-ragazzi This comes from another discussion. Matt will announce this on the mailing lists - we'll see if it sticks!

[cogmission]

@fergalbyrne this may not be germane at this point, but I would be happy to whip up a Java version (pretty sure I could sit down and do it in one session)? Otherwise my contribution is pretty much finito (after any design discussions I may participate in). :-/

Sent from my iPhone

On May 6, 2015, at 12:32 PM, Subutai Ahmad notifications@github.com wrote:

@david-ragazzi This comes from another discussion. Matt will announce this on the mailing lists - we'll see if it sticks!

—
Reply to this email directly or view it on GitHub.

[david-ragazzi]

@david-ragazzi This comes from another discussion. Matt will announce this on the mailing lists - we'll see if it sticks!

oohh! I look forward for this!!

[cogmission]

@david-ragazzi Have you been following the "Re: Cortical Learning Algorithm (CLA)" thread? Fergal suggests a slight change (may be more than slight in effect, though) to the CLA algorithm. I remember him talking about this back when I didn't know what a cortical column was! :-P But, now it seems like he's going to get to try it out.

NOTE:
I think the degree of depolarization should be kept on a "column object" (or indexed into the Connections object as a column variable - in Python NuPIC terms) - rather than trying to separately store the value on an individual cell. This way we keep the uniform feed-forward aspect of the single proximal dendrite; and we lessen the processing and looping.

@fergalbyrne Also, the convo @david-ragazzi and I were having was more about the eventual structure of NuPIC than about this issue specifically. I wouldn't want to combine the refactoring/restructuring of core components in this effort either. 👍

[rhyolight]

Ok guys, https://github.com/nupic-community/community.research has a decent README and is open for business. Contact me or another nupic-community owner for access.