Support options to compare in CheckpointHander

[Nic-Ma]

🚀 Feature

Hi @vfdev-5 ,

In current ignite CheckpointHander, it compares priority(may be metrics) > previous priority and save models:
https://github.com/pytorch/ignite/blob/master/ignite/handlers/checkpoint.py#L357
But we received several requests to customize this compare logic, is it possible to make options for it?

Thanks.

[vfdev-5]

Hi @Nic-Ma , thanks for the idea! What kind of customizations it is about:
e.g. change the sign of comparision, e.g. self._saved[0].priority > priority or something more complex ?

Let us think about what can be done.

[Nic-Ma]

Maybe I think the easiest way is to support a callable function with a default lambda?

compare_fn: Callable = lambda (new, old): new > old

Thanks.

[sdesrozis]

@Nic-Ma What should be the arguments of this callable ? I mean, are priority values good enough ? Or maybe it could be useful to have an extra argument, for instance the checkpoint object ?

[Nic-Ma]

HI @sdesrozis ,

Thanks for your support and good idea.
Could you pass engine as extra arg to the callable? Just pass the engine from Checkpoint.__call__(self, engine) to this callable? Users can use engine.state as a data-sharing object.
For example, maybe I can store the average value of last 10 validation metrics into engine.state.average_metric in another handler, then compare current metric with it to save model:

compare_fn: Callable = lambda (new_priority, old_priority, engine): new_priority > engine.state.average_metric

Thanks.

[vfdev-5]

A bit tricky thing to keep in mind is that self._saved[0].priority is sorted such that best model is the first. If we enable another rule, this should be also taken into account.

Anyway, I think this feature could be nice to have in ignite. @Nic-Ma would you like to send a draft PR for that ?

[Nic-Ma]

@vfdev-5 @sdesrozis , thanks for your support here.
Sure, I can try to make a PR next week if you guys are busy with other development tasks.

Thanks.

[Nic-Ma]

Hi @vfdev-5 and @sdesrozis ,

I tried to re-think this feature again, actually, we should do complicated comparing logic in customized score_function, and you guys already passed engine to this API.
So only need to add the option to support "greater_or_equal" instead of only "greater", I submitted PR #1597 for it.

Thanks.