Set alpha value to 0 for RunningAverage

[VinhLoiIT]

❓ Questions/Help/Support

When I look at the source code of RunningAverage class, I see that there is a factor variable, alpha, which is set to 0.98 as default. As far as I know, this is to smooth the value of a metric like this:

def compute(self) -> Union[torch.Tensor, float]:
    if self._value is None:
        self._value = self._get_src_value()
    else:
        self._value = self._value * self.alpha + (1.0 - self.alpha) * self._get_src_value()
    return self._value

The use case is that I want to log my metrics during validation time (using contrib.metrics.ProgressBar), these metrics should not be smoothed, however. One more case is that the smoothed value when plotting to Tensorboard is smoothed once again, which leads to quite unreliable the plot. By disabling the smooth factor, I set alpha to 0 and it raised an exception:

if not (0.0 < alpha <= 1.0):
    raise ValueError("Argument alpha should be a float between 0.0 and 1.0.")

I wonder why it is not 0.0 <= alpha <= 1.0 ? Is it a mistake or any idea behind it?

[vfdev-5]

@VinhLoiIT RunningAverage is mostly used to average values during the training: batch loss, etc
When setting alpha to 0, there is no point to use RunningAverage, so just skip it.

The use case is that I want to log my metrics during validation time (using contrib.metrics.ProgressBar)

Could you please detail your use case, such I could help better with the problem.

PS:

One more case is that the smoothed value when plotting to Tensorboard is smoothed once again

It is also possible to remove TB smoothing on the UI.

[VinhLoiIT]

The use case is that I use RunningAverage while validating, because it will trigger my metric computation every iteration (which can then pass to the tqdm logger ProgressBar class), and it supports epoch bound. Therefore, by setting alpha to zero, I would have the raw value of my metric while iterating through the validation set but still have the correct average at the end of the phase

The problem with EpochMetric class is that it saves the whole output of the engine, or Average class does not handle the last batch right if len(dataloader) is not divisible to batch_size

[vfdev-5]

The use case is that I use RunningAverage while validating, because it will trigger my metric computation every iteration

Normally, it should not trigger metric's computation.

OK, I think I understand your use-case, probably it would be better to do like that:

import time

import torch
from ignite.engine import Engine, Events
from ignite.metrics import Accuracy
from ignite.contrib.handlers import ProgressBar


acc = Accuracy()
sample_acc = Accuracy()

num_iters = 10
batch_size = 4
num_classes = 5
y_pred = torch.rand(num_iters, batch_size, num_classes)
y = torch.randint(0, num_classes, size=(num_iters, batch_size)).long()


def eval_fn(e, b):
    time.sleep(0.5)
    i = e.state.iteration - 1
    return y_pred[i, ...], y[i, ...] 


evaluator = Engine(eval_fn)

acc.attach(evaluator, "Accuracy")


@evaluator.on(Events.ITERATION_COMPLETED)
def compute_samplewise_metric(e):
    sample_acc.reset()
    sample_acc.iteration_completed(e)
    sample_acc.completed(e, "Samplewise Accuracy")

    
ProgressBar(persist=True).attach(evaluator, metric_names=["Samplewise Accuracy", ])

data = list(range(num_iters))
state = evaluator.run(data)
print(state.metrics)

What do you think ?

Average class does not handle the last batch right if len(dataloader) is not divisible to batch_size

Could you please raise a new issue on that. Thanks!

[VinhLoiIT]

First of all, let me show you my case:

I have written my own custom metric base on class Metric as in this tutorial.

@reinit__is_reduced
def reset(self) -> None:
    self._ed = 0.0
    self._num_examples = 0

@reinit__is_reduced
def update(self, output) -> None:
    .... # calculate the metric here..
    self._ed += distances.sum()
    self._num_examples += batch_size

@sync_all_reduce("_ed", "_num_examples")
def compute(self):
    if self._num_examples == 0:
        raise NotComputableError('CharacterErrorRate must have at least one example before it can be computed.')
    return self._ed / self._num_examples

So at a glance, this is similar to Accuracy class.

When I attach to the evaluator, it would call the Metric's attach method right?

CER(output_transform=lambda output: output[2:]).attach(evaluator, 'CER')

So when I look at the Metric's attach method:

def attach(self, engine, name):
    engine.add_event_handler(Events.EPOCH_COMPLETED, self.completed, name)
    if not engine.has_event_handler(self.started, Events.EPOCH_STARTED):
        engine.add_event_handler(Events.EPOCH_STARTED, self.started)
    if not engine.has_event_handler(self.iteration_completed, Events.ITERATION_COMPLETED):
        engine.add_event_handler(Events.ITERATION_COMPLETED, self.iteration_completed)

It call the self.completed and self.started once every epoch and self.iteration_completed every iteration. The problem is that the engine.state.metrics[name] = result line (in self.completed) is only get called when the epoch end, so the ProgressBar can not recognize it during the evaluating, it raised:

UserWarning: Provided metric name 'Loss' is missing in engine's state metrics: []  "in engine's state metrics: {}".format(name, list(engine.state.metrics.keys())))
UserWarning: Provided metric name 'Accuracy' is missing in engine's state metrics: []  "in engine's state metrics: {}".format(name, list(engine.state.metrics.keys())))

PS: it would produce similar warning to Loss and Accuracy class if I do as below, and the metric not show at the first epoch:

Loss(criterion).attach(trainer, 'Loss')
Accuracy().attach(trainer, 'Accuracy')

Hence, instead of rewriting the attach method, which is just to change the event of self.completed from Events.EPOCH_COMPLETED to Events.ITERATION_COMPLETED, I think it would be nicer to write:
RunningAverage(MyMetric(output_transform), alpha=0).attach(...)
First, its attach had been written to exactly what I want. Second, it is more readability (but still not equivalent to Average though). Furthermore, does not set alpha = 0 to turn off smoothing is common?

Thank you!

[vfdev-5]

@VinhLoiIT yes, your reasoning is correct about how Accuracy and Metric are working.

Maybe we can think to provide a specific interface for such iteration-wise metrics.
cc @justusschock @ykumards any thoughts ?
Personnally, I'm not favor for enabling RunningAverage(MyMetric(output_transform), alpha=0).attach(...)

PS: it would produce similar warning to Loss and Accuracy class if I do as below, and the metric not show at the first epoch.

This is OK as metrics computation in general use-case is to compute single value over a list of batches and not of a single batch.

[justusschock]

I think, we should definitely have an interface for that. I also agree that if you set alpha to 0, you don't have a running average any more.

Instead we should probably have a mixin class that handles that and should derive RunningAverage from this class

[VinhLoiIT]

I personally think that RunningAverage is fine, just interpret alpha parameter as a smoothing factor, so that it can be in range of [0..1]. alpha=0 means I do not want to smooth then it should return the raw value of the metric.

[vfdev-5]

I think, we should definitely have an interface for that. I also agree that if you set alpha to 0, you don't have a running average any more.

@justusschock I thought about adding an option to Metric.attach, something like this : #644 or just put an option for batch-wise computation...

[justusschock]

@vfdev-5 Sounds good as well

[VinhLoiIT]

I think, we should definitely have an interface for that. I also agree that if you set alpha to 0, you don't have a running average any more.

@justusschock I thought about adding an option to Metric.attach, something like this : #644 or just put an option for batch-wise computation...

@vfdev-5 I think it worth to mention that the new metric (e.g., Running) should have an option to reset the metric very reset_interval (similar to the behavior of running_loss in pytorch tutorial), so it can be used in both training and validating. Therefore, to calculate running metrics while training or validating, we can use Running class to get raw value and RunningAverage to get smoothed metric value. What do you think?