Easy way to run evaluation every n-th epoch?

[JohnGiorgi]

Hi!

Currently, if a validation or test dataset is provided, evaluation occurs at the end of each epoch during training. I am wondering if there is an easy way to modify this so that evaluation only occurs after every n-th epoch (or something like that).

The motivation is that my evaluation takes much longer than a training epoch, so I would rather it not run after each epoch, especially in situations like hyperparameter tuning.

I think I could get this behaviour by setting batches_per_epoch to the total number of batches I would like to train for, and then setting num_epochs to 1, but this feels like a bit of a hack.

[epwalsh]

I have a similar feature request, where I'd like to skip validation for the first few epochs since I know the performance will be ready bad early on in training. Changing batches_per_epoch is not ideal for this use-case since that effects the frequency of checkpoints later in on training, which I'd like to avoid.

We could probably tackle both of these cases by adding a flag in the trainer _should_validate_this_epoch: bool = True or something, which could be modified by a TrainerCallback each epoch.

[JohnGiorgi]

Okay, I think I am going to try to take this on if there isn't already someone working on it. My only concern is what this does to learning rate schedulers that depend on "validation_metric". I imagine an error would be raised somewhere on the first epoch that isn't followed by validation. Would probably need to catch this and raise an informative error message.

[JohnGiorgi]

This also applies to the "patience" parameter of GradientDescentTrainer, I believe.

[JohnGiorgi]

Addressed in #5534.