Example of Using L2 Regularizer

[cancan101]

Currently none of the examples show using the L2 regularizer.

An example of using it in a model would be helpful.

[benanne]

The current code for regularization was always intended as a placeholder. For now the easiest thing to do is just to construct the regularization terms manually. I don't think what we have right now is worth documenting, it needs to be thoroughly thought through and reimplemented first.

[cancan101]

Other than dropout and early-stopping, what other forms of regularization are currently implemented in Lasagne?

[benanne]

I don't think we have any code for early stopping in the library right now, there's no code for training loops yet. I guess @dnouri may have implemented this as part of his Lasagne extensions in https://github.com/dnouri/nolearn though.

Personally I usually stick to dropout and not much else, which is why that's all there is for now. Are you looking for anything in particular? As long as you can implement it in Theano, you can use it with Lasagne :)

[dimatura]

The regularization does work though. If your final output layer is l_out, then reg = lasagne.regularization.l2(l_out) returns an expression for the L2 norm of all the non-bias parameters in your network. If loss is a symbolic theano expression for your loss (e.g. cross entropy), then a regularized objective would be loss + 0.0001*reg. Then you can use gradient on this as usual.

[saskra]

I don't think we have any code for early stopping in the library right now

Did this change by now/ will it change someday?

[f0k]

Did this change by now/ will it change someday?

No / probably not. There are some useful discussions in #756, but there was no fully satisfying implementation at the time, and now there's hardly anyone to work on or review this. Writing a training loop for a specific experiment is easy enough, and writing a general training loop that covers all use cases and strikes the right balance between user-friendliness and transparency is very difficult. I'd rather have no training code at all than an incomplete solution.