make metric configurable that triggers checkpoint writing

[juliakreutzer]

Introducing a third configurable metric:

1. eval_metric: validation metrics like BLEU or ChrF; measured on the validation set only
2. schedule_metric: decides when the scheduler decreases the learning rate or stops learning; might be loss or ppl, might not match eval_metric (since often smoother curve)
3. new: ckpt_metric: decides when the model parameters are saved, i.e., a checkpoint is written; default is eval_metric

Before, checkpoint saving was coupled to eval_metric (new default), but potentially not matching schedule_metric (e.g., validating models with BLEU and storing when highest reached, but adapting learning rate according to loss). Now, one can decide whether all three align or are individually set.

[bastings]

 # if we schedule after BLEU/chrf, we want to maximize it, else minimize
        scheduler_mode = "max" if self.schedule_metric == "eval_metric" \
            else "min"

Does this decide to use min in case of loss/perplexity?
(And should it be configurable?)

[juliakreutzer]

Yes, this is still like it was done before. For now we only implement external evaluation metrics that should be maximized (BLEU, ChrF), so at the current state we don't have to make it configurable, I think.

[bastings]

I think to prevent bugs in the future maybe we should do the min/max switch based on the content of the setting, not based on if it is the same as eval_metric? What do you think?

[juliakreutzer]

What do you mean by content of the setting? All our possible options for "eval_metric" have to be maximized.

[bastings]

I mean if we allow any metric for that in the future that needs minimization, then that code will break.
I think we should set min/max as a function of BLEU/chrF/ppl/loss.

[bastings]

Let's leave this is a future improvement