Weight Decay

[cancan101]

Add support for weight decay to nolearn.

It looks like lasagne has support although it might need some more work.
Also see cuda-convnet docs.

[cancan101]

I think the loss function will need to have its signature updated to support this.

[dnouri]

Yes, I came across this myself recently; wanted to implement a cost function that depends on weights, so like an L2. Maybe it could be called objective, and the loss be deprecated. Maybe the arguments should be something like (layers, X_batch, y_batch), with layers being the output of self.get_all_layers()?

[cancan101]

This is what pylearn2 does:

layer_costs = [ wrapped_layer_cost(layer, coeff)
   for layer, coeff in safe_izip(model.layers, self.coeffs) ]

so it looks like self.get_all_layers() is likely enough.

[dnouri]

We should try and use whatever Lasagne comes up with as part of fixing its regularization story. See Lasagne/Lasagne#14