-
Notifications
You must be signed in to change notification settings - Fork 439
Add support for validation sets #85
Comments
The validation set support is a good idea. One concern I have is to how to fit with sklearn interface when a user wants to pass his specific validation set. Because usually you split your dataset 3 ways - train, validation and test and use validation for hyperparameter search. |
I made it as delegate validation function passing through fit(..., cross_valid_fn=my_func()). |
I was going to take a look at this (though I'm still learning my way around skflow). I see an early_stopping_rounds argument in the estimators (TensorFlowEstimator and its derived classes), and the argument is passed to the TensorFlowTrainer, which appears to implement early stopping logic. Is the current early stopping logic different from what's suggested in this issue? I'll pursue this issue further if I can understand how it differs from the current early stopping. |
@dansbecker Thanks for taking a look! Right now early stopping is done on training loss - e.g. if training converged, model stops before a number of required steps. On the other hand, using validation set is another option. But so far, in examples we were implementing it this way: https://github.com/tensorflow/skflow/blob/master/examples/resnet.py#L148 |
Thanks @ilblackdragon. That makes sense. On the issue of using the same data for early stopping and for hyperparameter search: Three options come to mind.
Thoughts? |
@ilblackdragon :Those three above are in addition to the one you mentioned My inclination is towards either your suggestion, or 2 or 3 in the note above. |
It would be nice if skflow had some support for validation sets to be used for early stopping and monitor validation set loss during training. This could be realized failry easily by adding a
fraction_validationset
to the TensorFlowEstimator. Withinfit
, the given training set could then be split into two parts.The text was updated successfully, but these errors were encountered: