Conditional metrics ?

[sdesrozis]

❓ Questions/Help/Support

How do you deal the confusion matrix metric ? It is an expensive metric that is interesting to have punctually during the training. A naive way is to attach ConfusionMatrix to evaluators but it could have a performance issue. Otherwise, a dedicated evaluator could be created but it multiplies evaluators... Any advice on that point ? My feeling is we need "conditional" metrics triggered wrt to specific predicat.

[vfdev-5]

@sdesrozis I think this depends on application and the additional cost needed to be quantified.
You can take a look at this example: https://github.com/pytorch/ignite/blob/master/examples/references/segmentation/pascal_voc2012/code/scripts/common_training.py#L137
Where validation IoU metric is computed with CM and evaluation is triggered every n epochs.
HTH

[sdesrozis]

Ok, thank you for your answer 👍
I noticed that because I did a classification training (approx 20 classes) and I had to move CM to postprocess due to performance. Tho, it wasn't a blocking issue.
In your example, how would you add additionnal metrics running at each epoch completed ? I don't see any other solution than adding an evaluator.

[vfdev-5]

In your example, how would you add additionnal metrics running at each epoch completed ? I don't see any other solution than adding an evaluator.

Yes, actually, there is no simple solution without adding another evaluator.
We can think of this as a feature request :)

[sdesrozis]

It would be a pleasure for me to contribute on that point 😊

[vfdev-5]

That's good ! So, now we need to decide on the API for such feature. Something simple and looking cool :)
For example, how about that:

trainer = ...
evaluator = ...
metrics = {
   "Accuracy": Accuracy(),
   "CM": ConfusionMatrix(num_classes=20)
}

@trainer.on(Events.EPOCH_COMPLETED)
def run_fast_evaluation(_):
    metrics["CM"].disable()  # metric CM is not computed
    evaluator.run(test_dataloader)


@trainer.on(Events.EPOCH_COMPLETED(every=5))
def run_extensive_evaluation(_):
    metrics["CM"].enable()
    evaluator.run(test_dataloader)

...

trainer.run(train_dataloader, max_epochs=100)

Obviously, with such solution, when logging the values from evaluator.state.metrics on every evaluator's Events.COMPLETED, accuracy will be updated every training epoch, but CM will be updated by steps, like that

^ CM 
|                     * * * * *
|           * * * * *
| * * * * *
|-------------------------------> epochs

which is not that good. To handle, that, probably, metrics["CM"].disable() should remove the key from evaluator.state.metrics, which can be dangerous (?).

Thougths ?
cc @justusschock @ykumards

[sdesrozis]

If CM is bot computed, dict metrics shouldn’t contain key « CM ». So easily checkable for logging. My question is should we disable explicitly metric or should we apply a predicat or filter to enable / disable implicitly during event handling.

[vfdev-5]

If CM is bot computed, dict metrics shouldn’t contain key « CM »

Correct, the state will be reset on each run.

Metrics are "attached" (they add multiple handlers), so, currently, there is no simple API to enable/disable a metric. I agree with you that we can also propose a sort of detach method that can remove all previously attached handlers. Maybe this is a better API than previously proposed disable/enable.

In my previous idea, it was thought as internal redirection of iteration_completed and completed methods. It is more simple to implement etc, but maybe it gives a not that clear API.

So, let's think about the following API:

trainer = ...
evaluator = ...
metrics = {
   "Accuracy": Accuracy(),
   "CM": ConfusionMatrix(num_classes=20)
}

for name, metric in metrics.items():
    metric.attach(evaluator, name)

metrics["CM"].detach()

@trainer.on(Events.EPOCH_COMPLETED)
def run_fast_evaluation(_):    
    if trainer.state.epoch % 5 == 0:
        metrics["CM"].attach(evaluator, "CM")

    evaluator.run(test_dataloader)

    if trainer.state.epoch % 5 == 0:
        metrics["CM"].detach()

...

trainer.run(train_dataloader, max_epochs=100)

EDIT: reworked handlers to avoid 2 eval runs.

[sdesrozis]

I like this API which is simple to deploy. An alternative is to try to have only one call to evaluator.run() (cheaper, less data load) and use the filter (every=5) attached to the metric itself. It shouldn’t solve logging issues (we have to know what is computed). Sorry I can’t propose snippets atm because my only access to github is my smartphone. Tho, I don’t know if it’s feasible neither user friendly...

[sdesrozis]

Nb : detach should have an argument evaluator because it shouldn’t be a collective detachment, should it?

Maybe the condition trainer.state.epoch % 5 == 0 could be managed as an extra argument of the first attachment loop (only for CM metric). I try to illustrate tomorrow (I’m in France). I noticed your remark about API clearness. I just investigate an option, hope it’s ok for you.

[vfdev-5]

Nb : detach should have an argument evaluator because it shouldn’t be a collective detachment, should it?

Technically with the current implementation, yes. But I think now we need to go futher with Metric class and register as a weakref the engine. As we'll allow attach/detach we need to make sure to handle properly multiple faulty attaches etc. So, once metric knows its engine, we wont need to insert engine as argument to detach.

Maybe the condition trainer.state.epoch % 5 == 0 could be managed as an extra argument of the first attachment loop (only for CM metric). I try to illustrate tomorrow (I’m in France)

I think this is tricky to do as attach uses evaluator, but event filtering is on trainer. But anyway if you have a clear idea how to do it, why not. Actually, me too in France, oui, il faut aller dormir :)

I just investigate an option, hope it’s ok for you.

I'm OK with that.

[sdesrozis]

Technically with the current implementation, yes. But I think now we need to go futher with Metric class and register as a weakref the engine. As we'll allow attach/detach we need to make sure to handle properly multiple faulty attaches etc. So, once metric knows its engine, we wont need to insert engine as argument to detach.

I understand that the metric could encapsulate the engine. But a metric could be related with several engines. Maybe I didn’t catch your idea.

Edit : a metric is used by several evaluators but one by one. Ok I understand 😉 (I hope)

I think this is tricky to do as attach uses evaluator, but event filtering is on trainer.

It looks like decoration pattern in your implementation.

Actually, me too in France, oui, il faut aller dormir :)

Super :)

[sdesrozis]

I implemented the following test

class DummyMetric(Metric):
    def __init__(self, name, output_transform=lambda x: x):
        self._name = name
        self.call_count_reset = 0
        self.call_count_compute = 0
        self.call_count_update = 0
        super(DummyMetric, self).__init__(output_transform=output_transform)

    def reset(self):
        print(f"{self._name} {id(self)} : reset")
        self.call_count_reset += 1

    def compute(self):
        print(f"{self._name} {id(self)} : compute")
        self.call_count_compute += 1

    def update(self, output):
        print(f"{self._name} {id(self)} : update")
        self.call_count_update += 1


def test_metric_filtering():

    trainer = Engine(lambda e, b: None)
    evaluator = Engine(lambda e, b: None)

    # metrics creation
    # Remark: reset() is called at creation
    dummy1 = DummyMetric('dummy1')
    dummy2 = DummyMetric('dummy2')

    # dummy2 is attached
    dummy2.attach(evaluator, 'dummy2')

    def dummy1_filter():
        value = trainer.state.epoch % 2
        print(f"trainer epoch {trainer.state.epoch} : "
              f"value={value}, "
              f"filter={value == 0}")
        return value == 0

    dummy1_on_the_fly = dummy1.attach_on_the_fly(evaluator, 'dummy1', dummy1_filter)

    @trainer.on(Events.EPOCH_COMPLETED)
    @dummy1_on_the_fly
    def _(_):
        # decoration on the fly
        # if filter ok, dummy1 is attached on the fly
        #if dummy1_filter():
        #    dummy1.attach(evaluator, 'dummy1')

        evaluator.run([0, 1, 2])

        # decoration on the fly
        # dummy1 is now detached
        #if dummy1_filter():
        #    dummy1.detach(evaluator)

    trainer.run([0, 1, 2, 3, 4, 5], max_epochs=5)

    print(f"dummy1 : reset={dummy1.call_count_reset} "
          f"compute={dummy1.call_count_compute} "
          f"update={dummy1.call_count_update}")
    print(f"dummy2 : reset={dummy2.call_count_reset} "
          f"compute={dummy2.call_count_compute} "
          f"update={dummy2.call_count_update}")

    assert(dummy1.call_count_reset == 3) # Remark: compute + 1 due to creation
    assert(dummy1.call_count_compute == 2)
    assert(dummy1.call_count_update == 6)

    assert(dummy2.call_count_reset == 6)
    assert(dummy2.call_count_compute == 5)
    assert(dummy2.call_count_update == 15)

I added following in Metrics

  def detach(self, engine: Engine) -> None:
      if engine.has_event_handler(self.completed, Events.EPOCH_COMPLETED):
          engine.remove_event_handler(self.completed, Events.EPOCH_COMPLETED)
      if engine.has_event_handler(self.started, Events.EPOCH_STARTED):
          engine.remove_event_handler(self.started, Events.EPOCH_STARTED)
      if engine.has_event_handler(self.iteration_completed, Events.ITERATION_COMPLETED):
          engine.remove_event_handler(self.iteration_completed, Events.ITERATION_COMPLETED)

  def attach_on_the_fly(self, engine: Engine, name: str, filter_fn: Callable) -> Callable:

      def decorator(f: Callable) -> Callable:

          def fun_(*args, **kwargs):
              if filter_fn():
                  self.attach(engine, name)
              f(*args, **kwargs)
              if filter_fn():
                  self.detach(engine)

          return fun_

      return decorator

@vfdev-5 any advice ? I didn't add engine ref in Metrics because metrics are implemented stateless atm.

[sdesrozis]

Alternative using custom_event :

 def dummy1_filter(engine, event):
      value = engine.state.epoch % 2
      print(f"engine epoch {engine.state.epoch} : "
            f"value={value}, "
            f"filter={value == 0}")
      return value == 0

  @trainer.on(Events.EPOCH_COMPLETED(event_filter=dummy1_filter))
  def _(_):
      dummy1.attach(evaluator, 'dummy1')

  @trainer.on(Events.EPOCH_COMPLETED)
  def _(_):
      evaluator.run([0, 1, 2])

  @trainer.on(Events.EPOCH_COMPLETED(event_filter=dummy1_filter))
  def _(_):
      dummy1.detach(evaluator)

Question is : should we avoid explicit calls to attach/detach ? It seems to be pre/post action of events.

[vfdev-5]

@sdesrozis thanks for the snippets. I'll take a look in details a bit later. In general, I would say
explicit calls will be more intuitive. Solution with attach_on_the_fly looks interesting but probably the API needs to be polished.