Metrics and EvalProtocol API

[AntonioCarta]

Metrics and EvalProtocol are a little bit unclear to me.

• What is EvalProtocol's job? Most of the code implements Tensorboard logging operations but the name hints to something more than that.
• Right now metrics do not have a uniform API, and each one takes different argument for the compute method. Each time we add a new metric, we also have to add a new if case inside EvalProtocol's get_results.

I would prefer a generic EvalProtocol that controls printing and logging and only delegates the computations to the metrics (e.g. instead of printing inside compute EvalProtocol calls the __str__ method). I would also prefer to be able to choose where to print the metrics (output file, tensorboard, stdout).

[vlomonaco]

Yes, that would be ideal!

[AntonioCarta]

Actually, it was more of a question than a proposal. I still don't have a clear idea on how to solve this problem.

[vlomonaco]

I think the EvalProtocol job should be exactly what you said:

I would prefer a generic EvalProtocol that controls printing and logging and only delegates the computations to the metrics (e.g. instead of printing inside compute EvalProtocol calls the str method). I would also prefer to be able to choose where to print the metrics (output file, tensorboard, stdout).

As for the metrics, as soon as we have all the the metrics implemented we can define an unique signature for the compute method... let's wait a little on this!

[akshitac8]

@vlomonaco Can I work on this and also it would be helpful if you could suggest some initial metrics that are needed and also is issue #51 solved?

[AntonioCarta]

#51 is closed.

Regarding the Metrics, maybe we can use Flows to define how and when to compute each metric? The problem that we have right now is that each metrics requires different computations and arguments. This could be easily solved with a callback system, like the flows used for training and test.

Take as an example the memory usage MU. It should be printed only once, instead it is printed everywhere because the EvaluationProtocol does not know how to print it:

Training completed
Computing accuracy on the whole test set
Task 0 - CF: 0.0000
Train Task 0 - MU: 0.259 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.002227343698385049; Avg Acc 0.9242105484008789
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.265 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.06294100686298648; Avg Acc 0.0
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.265 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.0536595421147698; Avg Acc 0.0
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.268 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.05200100937101282; Avg Acc 0.0
Task 0 - CF: 0.9242
Train Task 0 - MU: 0.270 GB
Confusion matrix, without normalization
[Evaluation] Task 0: Avg Loss 0.06933177224355726; Avg Acc 0.0
Start of step  1

Each metric will probably need to implement a couple of methods, while the others will stay empty, so the implementation of new metrics should not be more complex than it is right now.

[vlomonaco]

Hi @AntonioCarta you are totally right. Indeed now the metrics are called through the EvaluationPlugin!

Being it a plugin, it can implement all the callbacks independently from the main strategy (all the plugin methods will be called before the main strategy methods). This also means that the calls he make can be fine-tuned to specific metric needs. Does it make sense or am I missing something?