--valid-metrics-stop-on [first|all|any]

[emjotde]

Feature description

Add an option --valid-metrics-stop-on [first|all|any] that defines together with --early-stopping N and --valid-metrics metric1 ... metricN if early stopping should take into account only first, all, or any metrics.

• first: current default behaivor, stop if first metric stalled N times
• all: stop if all of the metrics have currently stalled N or more times (if the minimum of all stalled counts is N)
• any: stop if any of the listed metric has stalled N times (if the maximum of all stalled counts is N).

Example

• --valid-metrics perplexity chrf bleu --early-stopping 10 --valid-metrics-stop-on all: training stops if chrf stalled 10 times, perplexity 15 times and bleu 11 times, but not if e.g. perplexity stalled only 9 times with rest as described.

[emjotde]

Maybe we want more complicated pattern like --valid-metrics-stop-on 0 1 any which ignores the third metric? But maybe that's overkill.

[snukky]

It's been implemented. Closing.