Transforming metric values

[failable]

❓ Questions/Help/Support

The document of Metircs says that return value of compute can be Any. So I'm trying to do this in a single pass evaluation over the whole validation data loader.

class SuperMetrics(Metric):

  def __init__(self, num_labels, output_transform=lambda x: x, device=None):
      self.num_labels = num_labels
      self._y = None
      self._y_pred = None
      self._num_drugs = None
      super(SuperMetrics, self).__init__(output_transform=output_transform,
                                         device=device)

  def compute_metrics(self, y, y_pred):  # pylint: disable=no-self-use
      y = y.toarray()
      y_pred = y_pred.toarray()
      y[y > 0] = 1
      y_pred[y_pred > 0] = 1

      hamming_loss = metrics.hamming_loss(y, y_pred)
      macro_f1 = metrics.f1_score(y, y_pred, average='macro')
      macro_precision = metrics.precision_score(y, y_pred, average='macro')
      macro_recall = metrics.recall_score(y, y_pred, average='macro')
      micro_f1 = metrics.f1_score(y, y_pred, average='micro')
      micro_precision = metrics.precision_score(y, y_pred, average='micro')
      micro_recall = metrics.recall_score(y, y_pred, average='micro')

      return {
          'hamming_loss': hamming_loss,
          'macro_f1': macro_f1,
          'macro_precision': macro_precision,
          'macro_recall': macro_recall,
          'micro_f1': micro_f1,
          'micro_precision': micro_precision,
          'micro_recall': micro_recall
      }

  @reinit__is_reduced
  def reset(self):
      self._y = []
      self._y_pred = []
      self._num_drugs = []
      super(SuperMetrics, self).reset()

  @reinit__is_reduced
  def update(self, output):
      y, y_pred, num_drugs = output
      self._y += y
      self._y_pred += y_pred
      self._num_drugs += num_drugs

  @sync_all_reduce('_y', '_y_pred', '_num_drugs')
  def compute(self):
      num_examples = len(self._num_drugs)

      rows = []
      y_columns, y_pred_columns = [], []
      for i, (y_sample, y_pred_sample, num_drug_sample) in enumerate(
              zip(self._y, self._y_pred, self._num_drugs)):
          rows += [i] * (num_drug_sample - 2)
          y_columns += y_sample[1:1 + num_drug_sample - 2]
          y_pred_columns += y_pred_sample[:num_drug_sample - 2]
      values = [1] * len(rows)
      y = coo_matrix((values, (rows, y_columns)),
                     shape=(num_examples, self.num_labels))
      y_pred = coo_matrix((values, (rows, y_pred_columns)),
                          shape=(num_examples, self.num_labels))

      return self.compute_metrics(y, y_pred)

The problem is, if this metric is attached to the evaluator by passing metrics={'super': SuperMetrics(vocab_size))}, I will get a nested metric value of engine.state.metrics. This is fine if I only print it to the terminal though. But I can not figure out a way to make it work with NeptuneLogger.

neptune_logger.attach(
    evaluator,
    log_handler=OutputHandler(tag='val',
                              metric_names='all'),
    event_name=Events.EPOCH_COMPLETED(every=params['eval_freq']))

Is there a safe way and place to flatten engine.state.metrics? Or should I do this? Is there any advice to compute all this metrics once using ignite? Thanks!

[sdesrozis]

@Isolet Thank you very much for your question !

You can use MetricLambda to create proxies of your nested metric (and use it in loggers). For instance, have a look of the following example

class NestedMetric(Metric):
    def __init__(self):
        self.dict_ = {"acc": Accuracy(),
                      "pre": Precision(),
                      "rec": Recall()}
        super(NestedMetric, self).__init__()

    def reset(self):
        for _, v in self.dict_.items():
            v.reset()

    def update(self, output):
        for _, v in self.dict_.items():
            v.update(output)

    def compute(self):
        return {k: v.compute() for k, v in self.dict_.items()}


y_pred = torch.randint(0, 2, size=(10,)).long()
y = torch.randint(0, 2, size=(10,)).long()

m = NestedMetric()
m.update((y_pred, y))
m.update((y_pred, y))
m.update((y_pred, y))
print(m.compute())

def fn_acc(v):
    return v["acc"]

acc = MetricsLambda(fn_acc, m)

def fn_pre(v):
    return v["pre"]

pre = MetricsLambda(fn_pre, m)

def fn_rec(v):
    return v["rec"]

rec = MetricsLambda(fn_rec, m)

m.reset()
m.update((y_pred, y))
m.update((y_pred, y))
m.update((y_pred, y))

print(acc.compute())
print(pre.compute())
print(rec.compute())

We should include an example in the doc.

Please tell me if that does not answer to your problem.

HTH

[vfdev-5]

@Isolet looking into your implementation, it looks like you store all data _y, _y_pred, _num_drugs during the run and in the end compute the metrics. We have a similar helper class EpochMetric, however, due to _num_drugs it seems it wont fit your need.

Please, be aware, that in distributed data configuration, the following code

@sync_all_reduce('_y', '_y_pred', '_num_drugs')
  def compute(self):

will try compute the sum of _y, _y_pred, _num_drugs across all processes. This is, probably, not the expected result. Expected results are concats of these values. However, this can lead to memory overflow for long runs with large data stored in internal variables.

[failable]

@sdesrozis Thanks for the hint of implementing nested metrics! That would be quite helpful. However, in this specific case (require predictions over all batches), since the doc says updating nesting metrics will not update the nested metric, is there a way to maintain the only one copy y and y_pred in the nested metric instead of all nesting lambda metrics?

[failable]

@vfdev-5 Thanks for correcting me about sync_all_reduce. I just copied this piece of code from the document and have not completely dig into it ...😅 I switch from lightning to ignite yesterday and quite happy with ignite! About the memory issue, I also read from the document about the multi-label case before I implemented this 'SuperMetric', I can't find a better way right now...

[vfdev-5]

@Isolet thanks for the feedback :)

since the doc says updating nesting metrics will not update the nested metric, is there a way to maintain the only one copy y and y_pred in the nested metric instead of all nesting lambda metrics?

In order to "unneest" your computed metrics, I think you can simply override

ignite/ignite/metrics/metric.py

Line 138 in 0b36397

def completed(self, engine: Engine, name: str) -> None:

like that:

    def completed(self, engine: Engine, name: str) -> None:
        result_dict = self.compute()
        for name, result in result_dict.items():
            engine.state.metrics[name] = result

However, this can be also an enhancement for Metric class: if result is a dictionary -> write its items into engine.state.metrics instead of result as dict.
Any thoughts @sdesrozis ?

[sdesrozis]

However, this can be also an enhancement for Metric class: if result is a dictionary -> write its items into engine.state.metrics instead of result as dict.
Any thoughts @sdesrozis ?

I think It's a good improvement. However, we should check that dict contains tensors or scalars.

In the same way, we could imagine the same flattening feature for list

 def completed(self, engine: Engine, name: str) -> None:
        result_list = self.compute()
        for i, result in enumerate(result_list):
            engine.state.metrics[name + i] = result

[sdesrozis]

@Isolet Maybe you are interested to contribute on that feature ?

[failable]

@sdesrozis I might try it tomorrow. Shall we limit the type of return value of Metric.compute to numbers.Number, torch.tensor, List, and Dict? I'm not sure if I'm missing any use case of the original Any type?

[sdesrozis]

Very happy to have you with us on this issue 😊

ATM I don’t know if list is a good idea or not. I suggest working on dict first since it’s your needs.

Tell me if you need help on implementation, it would be a pleasure for me to help!! All the stuff is in the metric.py file 😉

[failable]

😅It took me some time to learn how to write tests... I've submitted a PR of this.