Cumulative accuracies

[AlbinSou]

Adds the cumulative accuracy metric from "On the importance of cross-task features in class-incremental learning" https://arxiv.org/abs/2106.11930.

For now I only put the accuracies, maybe later I will add the forgetting metric derived from this accuracy metric.

I don't know if we should change the name a bit or not, I am afraid that it could be confounded with Average Anytime Accuracy that I know some works also call it similarly (like cumulative or accumulative accurracy).

I tested it on my baselines branch and checked that it worked, I'm still thinking of a nice unittest I could add.

[coveralls]

Pull Request Test Coverage Report for Build 4936925152

• 74 of 148 (50.0%) changed or added relevant lines in 3 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.2%) to 73.86%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
avalanche/evaluation/metrics/cumulative_accuracies.py	54	128	42.19%

Totals
Change from base Build 4925589534:	-0.2%
Covered Lines:	15292
Relevant Lines:	20704

---

💛 - Coveralls

[AntonioCarta]

Hey Albin, thanks for the PR. Can you add some basic tests? Examples:

• check taht CA==0 when all the predictions are wrong
• check that CA==1 when all the predictions are correct
• check that the number of samples used is correct (i.e. not using samples from other tasks)
• check that all the tasks have an accuracy value

Just to be sure that the computation is working as expected.

[AlbinSou]

Yes, I will add some tests but also I'm trying to add the corresponding forgetting measure

[AlbinSou]

Added cumulative forgetting, I will add all the tests asap.

There is one thing I wanted to comment, the way the cumulative accuracies are reported now is the following:

CumulativeAccuracy/eval_phase/test_stream/Task000: 0.88
CumulativeAccuracy/eval_phase/test_stream/Task001: 0.69
...

However, this can be a bit confusing since the task ID is never changing in class-incremental learning (always 0), but I still want to show the per-experience CumulativeAccuracy. Is this the correct way to do so or should I change it ?

[AntonioCarta]

However, this can be a bit confusing since the task ID is never changing in class-incremental learning (always 0), but I still want to show the per-experience CumulativeAccuracy. Is this the correct way to do so or should I change it ?

I would call them by exp, like CumulativeAccuracy/eval_phase/test_stream/Exp000.
Can you change it? the rest looks ok.

[AlbinSou]

However, this can be a bit confusing since the task ID is never changing in class-incremental learning (always 0), but I still want to show the per-experience CumulativeAccuracy. Is this the correct way to do so or should I change it ?

I would call them by exp, like CumulativeAccuracy/eval_phase/test_stream/Exp000. Can you change it? the rest looks ok.

I changed it, I also prefer it.

[AntonioCarta]

cool, you just need to pull the master branch to fix the merge conflict.