Add callback so we can do pruning and check for nan values.

[PhilipMay]

I added a callback function. This way I can implement pruning and check for nan values when doing automated hyperparameter optimization. What do you think?

Some docstrings are still missing so if you agree I would complete the PR.

Thanks
Philip

[PhilipMay]

Do you have any questions about this PR?
I would really appreciate a merge.

[nreimers]

Hi,
thanks for adding this. I think is is a useful function and I will try to merge it into the next release.

Currently I try to think about if the callback could return something that influences the training, for example, stopping the training or what so ever.
But:

1. I am not sure what the best option would be to implement this.
2. Which features would be needed? Stopping training? Trigger to save / not save the model?
3. And if a return would be needed at all.

Also, it would be an option to add the callback to the evaluators instead of to the fit function. This would have the advantage that each evaluator could have it's own callback, which might (or might not) be quite useful if you evaluate your model on more than dev dataset.

It would be great to learn more how you are using this callback for hyperparameter tuning.

[PhilipMay]

It would be great to learn more how you are using this callback for hyperparameter tuning.

I use it in optuna for automated hyperparameter optimization. Optuna has something called pruning where
you can stop evaluating one set of hyperparameters when you see that it is unpromising.

I noticed that some hyperparameter combinations return nan at some point during training.
This is when I prune by raising a special "prune exception" in the callback.

It looks like this:

def callback(value, a, b):
    print('callback:', value, a, b)
    if math.isnan(value):
        raise optuna.exceptions.TrialPruned()

See more here: https://optuna.readthedocs.io/en/stable/tutorial/pruning.html

[nreimers]

Thanks for the PR. Merged it and it will be part of the next release (0.3.6)