whats the most idiomatic way to track the best validation accuracy and then fire events

[CDitzel]

Following the common pattern

@trainer.on(Events.EPOCH_COMPLETED)
    def log_validation_results(trainer):
        print(evaluator.run(evaluation_loader))  # returns state

an evaluator is instantiated once but its run method is called for one epoch every time the trainer has completed an epoch and the evaluator.state gets reset after every epoch.

What is the idiomatic way to fire a custom Event in case the current evaluation accuracy surpasses all evaluation accuracies before?

I need keep state of evaluators best accuracy across runs, but its state member is nulled upon calling run, i.e. a new epoch. Is the best way here to store this value within the engine itself or rather define a whole new class like its done in https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/custom_events.py

I read #627 but there the evaluator result is stored within the state of the trainer, which seems like an ugly hack

[vfdev-5]

What is the idiomatic way to fire a custom Event in case the current evaluation accuracy surpasses all evaluation accuracies before?

Maybe like this

• attach a handler to trainer to compute accuracy after each epoch (we still need this)
• attach a handler to evaluator to store best value in trainer.state. State according to our docs is intended to store any user data: https://pytorch.org/ignite/concepts.html#state
  • and call custom method "in case the current evaluation accuracy surpasses all evaluation accuracies before"

trainer = ...
evaluator = ...
# and attach Accuracy metric with name 'val_acc',  for example

@trainer.on(Events.EPOCH_COMPLETED)
def compute_validation_metric(_):
     evaluator.run(val_dataloader)

@trainer.on(Events.STARTED)
def init_best_metric_values(_)
     trainer.state.best_metrics['val_acc'] = 0.0

@evaluator.on(Events.EPOCH_COMPLETED)
def store_best_metric(_):
    metrics = evaluator.state.metrics
    if metrics['val_acc'] > trainer.state.best_metrics['val_acc']:
        trainer.state.best_metrics['val_acc'] = metrics['val_acc']
        custom_logic(...)

I thinks, evaluator's role is just to compute metrics without keeping history of previous computations. On the other hand, trainer can have such a role and store computed values and compare them...

What do you think ?

[CDitzel]

thank you for your detailed reply. I appreciate it. Thats one solution I came up with.
Another ist, that I wrote a custom class and after overloading its call operator, attached it to the event epoch.completed of the evaluator that keeps state and does all the work, much like ModelCheckpoint does.

by the way, is it advised to fire custom events in handler functions within client code or should those be restricted to library code? In other words, should decorated functions be allowed to fire custom events which have been registered before?

[vfdev-5]

In other words, should decorated functions be allowed to fire custom events which have been registered before?

Yes, definitely. One of examples we provide is bptt engine in contrib. A custom event is triggered in update function. But we can also imagine to trigger a custom event from any function.

[vfdev-5]

@CDitzel I close this issue as answered. Feel free to reopen if needed

[CDitzel]

one more question, coming from other languages like C++ etc., why would I ever need to have engine or trainer as parameter to the handler function instead of _ like you did above, when the trainer is in scope anyways?

[vfdev-5]

@CDitzel you can imagine that those handlers can be defined in a separate module where trainer/evaluator are not in the scope. Refactoring can make the code more clear after all.

[CDitzel]

thank you