PES learning rule on decoders

[tbekolay]

This is a first proof of concept of decoder learning in new Nengo. It just uses the PES rule to learn better decoders, but the way it's implemented is how all the other learning rules will have to be implemented, so I wanted to get feedback on the syntax before I make this work on weights and implement hPES, etc.

First off, here it is learning a communication channel super fast. It starts off with decoders that solve the function f(x) = -1, but does the communication channel after that.

Obviously a trivial example, but it's the same math as before so whatev's.

Check out learn_communicationchannel.py and see what you think of the syntax. It's just a new kwarg to the connect call (and therefore to the Connection classes).

Things I would appreciate feedback on:

1. Does it make sense to be a kwarg to connect? I'm thinking that the way we pass neuron types to ensembles should be the same way we pass learning rules to connect, but perhaps the semantics of that don't fit as nicely as I think they do.
2. Right now the PES class gets connection information by sneakily storing the connection it's part of. This was just easier, but I can definitely see why this could be confusing. Neurons, for example, don't keep a handle to the ensemble they're part of, and instead have functions that the ensemble calls. Perhaps learning rules should follow this? Or perhaps neuron types should get a handle to its ensemble? This made creating build_pes very easy, because build_connect doesn't know anything about learning rules. But we have to enforce that learning rules build after all connections. Perhaps the same could happen with neural nonlinearities, which would simplify the build_ensemble method?
3. This may only be true of the decoder PES rule, but I think that I could do it using a DotInc or something rather than making SimPES. Is that true? It might involve reshaping some signals to do it though, is it worth it?

[tcstewar]

Given the just-recently-completed discussion on changing the syntax of the API, what are you thinking of for this syntax? Will it be:

with model:
   # Create a modulated connection between the 'pre' and 'post' ensembles
   nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape),
              learning_rule=nengo.PES(error))

or will it be:

with model:
   # Create a modulated connection between the 'pre' and 'post' ensembles
   nengo.LearningConnection(pre, post, function=lambda x: -1 * np.ones(x.shape),
              learning_rule=nengo.PES(error))

or even:

with model:
   # Create a modulated connection between the 'pre' and 'post' ensembles
   nengo.PESConnection(pre, post, error, function=lambda x: -1 * np.ones(x.shape))

I think I'd lean towards the middle option.

I also don't quite understand the function argument. Is that going to be something that people have to specify all the time? Or is that just you initializing it to something weird for this one particular case?

How do people specify an initial weight matrix, or an initial random distribution of weights?

(Basically, I'm trying to think of the common use cases people want when they add in a learning rule)

[hunse]

Mixing objects and strings... my eyes, my eyes!!!

[tbekolay]

I was just initializing it to something weird, because otherwise it already does a communication channel. This is just the decoder level learning; in the weight learning case, it will be however we specify weight matrices for non-learning neuron-to-neuron connections (hence @hunse's question about making sure those work).

Right now, I'm proposing the top option. I did consider the two possibilities you presented, and in fact, originally used something like the middle one (but not quite). The reason I went with the one I went with is to A) make the use of neruons and learning rules essentially the same, where neurons apply to ensembles and learning rules apply to connections, and B) to reduce duplication of connection code, which should be unchanged except for the additional learning op that modifies the decoder / weight signals.

A few more possibilities:

with model:
   conn = nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape))
   pes = nengo.PES(conn, error)

with model:
   conn = nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape))
   pes = nengo.LearningRule(conn, nengo.PES(error))

The advantages of these two is that it is much cleaner when applying multiple learning rules to the same connection. This is one place where the analogy neuron type is to ensembles as learning rule is to connections breaks down, because you can only have one neuron type, but possibly multiple learning rules. We could do this by accepting a list for learning_rule, but that's a bit messy.

[jaberg]

Aren't there a significant variety of proposed and
possibly-worth-implementing learning mechanisms? What's the purpose or
sense of a "learning rule" in general?

I think the suggestion with nengo.PES(conn, error) is best because it's
short, non-committal regarding what nengo provides, and get's just as much
info from the user as anything else.

On Wed, Nov 13, 2013 at 11:59 AM, Trevor Bekolay
notifications@github.comwrote:

I was just initializing it to something weird, because otherwise it
already does a communication channel. This is just the decoder level
learning; in the weight learning case, it will be however we specify weight
matrices for non-learning neuron-to-neuron connections (hence @hunsehttps://github.com/hunse's
question about making sure those work).

Right now, I'm proposing the top option. I did consider the two
possibilities you presented, and in fact, originally used something like
the middle one (but not quite). The reason I went with the one I went with
is to A) make the use of neruons and learning rules essentially the same,
where neurons apply to ensembles and learning rules apply to connections,
and B) to reduce duplication of connection code, which should be unchanged
except for the additional learning op that modify the decoder / weight
signals.

A few more possibilities:

with model:
conn = nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape))
pes = nengo.PES(conn, error))

with model:
conn = nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape))
pes = nengo.LearningRule(conn, nengo.PES(error))

The advantages of these two is that it is much cleaner when applying
multiple learning rules to the same connection. This is one place where the
analogy neuron type is to ensembles as learning rule is to connections
breaks down, because you can only have one neuron type, but possibly
multiple learning rules. We could do this by accepting a list for
learning_rule, but that's a bit messy.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/202#issuecomment-28412286
.

[tcstewar]

I like what @jaberg is suggesting here. It may be very premature to talk about a generic LearningRule class, when we've only got one example of it and we know that different learning rules require very different backend implementations

I also really like the idea of initially making a connection exactly how you would normally, then adding a learning rule to it. That could work very well in teaching situations.

with model:
   conn = nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape))
   pes = nengo.PES(conn, error)

This is also the first time I've seen a use for keeping the Connection object around for later.

[tbekolay]

The nengo.LearningRule is to make it clear what PES is. Looking at this script alone doesn't tell you what nengo.PES does, but with nengo.LearningRule you should get it without having to turn to documentation. But I don't really have any strong opinions as to any of these options, at this point any of them are easy to implement.

[tcstewar]

Ooh, let me update my suggestion a bit:

with model:
   conn = nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape))
   nengo.PESLearning(conn, error)

[hunse]

@tcstewar : just what I was thinking 👍

[tcstewar]

(calling it PESLearning I think helps make it semi self-documenting)

[tbekolay]

Sorry, I didn't mean to introduce a generic LearningRule class. Consider this the first option instead.

with model:
   conn = nengo.Connection(pre, post, function=lambda x: -1 * np.ones(x.shape))
   model.learn(conn, nengo.PES(error))

or

conn = model.connect(pre, post, function=lambda x: -1 * np.ones(x.shape))
model.learn(conn, nengo.PES(error))

[hunse]

@tbekolay : we had dreamed at one point about being able to write a learning rule as a Python function in a script, so that the user can easily play around with different learning rules. Do you think this will be possible with the way you have things set up? Could it be done through a CustomLearningRule class? Or is there some better way to do it? Obviously this wouldn't be the type of thing that would be supported by all backends, but could we at least get it to work in the reference simulator, and maybe OCL, to make it easy to prototype learning rules? Should I add another issue for this?

[tbekolay]

Nah, this is a WIP so we can talk about learning rules in general here. The problem with that is the need to have a common interface for all learning rules, which is like, impossible. In Java, I split it into learning rules without error signals (unsupervised) and those with error signals (supervised) which seemed like a general split, and you could essentially ask people to provide the step function, given what the step function uses in PES. So, it's possible, but I think there are 4 cases: unsupervised decoder (encoder?) learning, unsupervised weight learning, supervised decoder learning, supervised weight learning. Is it useful to expose all those cases, and have PES / HPES be instances of those learning rule classes? I don't know if I'm convinced that that's useful...

[jaberg]

Hey guys, I'd like to revive this issue. I wrote a learning rule for learning a classifier. I wrote it to do some comparisons with less-neural machine learning algorithms, but I wonder if it might be interesting as a learning rule for the BG?

Anyway, I think I'd like to add it to this PR so that if others agree it would be useful in Nengo, then we can coordinate the two of them so they match.

First item of business though: rebasing this on master. Has anyone (@tbekolay) done this? I'll start with that.

[tbekolay]

I haven't rebased to master yet, so go nuts! Should it be part of the same PR though? Perhaps we could make it another PR but base that one on the PES branch? Or is that too hard to coordinate?

[jaberg]

@tbekolay Sounds good. I'll make the new rule a PR against the PES branch.

[studywolf]

What's the status with learning now? I saw the PR in #232, any movement on this since then?

[tbekolay]

I haven't worked on it, but @e2crawfo has, and has it working with Nengo OCL. #232 introduced some things in addition to the learning that makes it incompatible with the current codebase. I suspect that learning rules are going to need something similar to what's being discussed in #285, so perhaps I'll give this a rewrite in that style.

[youssefzaky]

I've been using the PES rule from the branch cleanup_learning, seems to work fine.

[studywolf]

Gahh, that branch isn't up to date with the current syntax though.

[arvoelke]

See #303.

[tbekolay]

Now superceded by other learning PRs, so closing.