Example for storing statistics from training

[tbrittoborges]

After studying the sknn api I could not plot a validation curve to test over-fitting.

from sknn.mlp import Classifier, Layer
nn = Classifier( layers=[ Layer("Maxout", units=100, pieces=2), Layer("Softmax")], learning_rate=0.001, n_iter=25)
result = nn.fit(X, y, w)

Which attribute carries the result mse/mcc per epoch/iteration?

[tbrittoborges]

I am working on it. To get started:

valid_err = []
train_err = []
def get_errs(**variables):
    valid_err.append(variables['best_valid_error'])
    train_err.append(variables['best_train_error'])

nn = Classifier(
    layers=[Layer("Softmax")], verbose=0, n_iter=5, callback={'on_epoch_start': get_errs}, valid_size=.2)

[alexjc]

Perfect, you found the callback I was going to suggest!

Is it working now?

[tbrittoborges]

It's working fine. I think it a bit obscure, thou.
One suggestion would have support to cross-validation, as in sklearn.

[alexjc]

A section in the documentation would help I guess.

Most sklearn APIs are supported. Which in particular would help?

[alexjc]

I quite like the look of this code. It's very explicit. I'll add it to the User Guide!

If you have more information about how you'd like to do cross-validation that isn't supported already, let me know and we can look into it separately in another issue!

valid_err, train_err = [], []
def get_errs(best_valid_error, best_train_error, **_):
    valid_err.append(best_valid_error)
    train_err.append(best_train_error)

nn = Classifier(
    layers=[Layer("Softmax")],
    n_iter=5, valid_size=.2, verbose=0,
    callback={'on_epoch_finish': get_errors})

[tbrittoborges]

I will look into embedding cross-validation in the Classifier.

The first line of the example needs a correction:
valid_err, train_err = [], []

[alexjc]

Thanks for the report and suggestion! I'm leaving this open until I update the docs.