Reproduce iCarl results

[AndreaCossu]

Link to the paper

[AntonioCarta]

This should be enough: https://github.com/ContinualAI/avalanche/blob/master/examples/icarl.py

[rudysemola]

I gladly try it

[qsunyuan]

Hi! sorry to interrupt you.

I reproduced iCarl results (default args), but I cannot achieve the trarget results of 0.62 in this link, even I tried some different random seed.

I would like to know where did the value of avearge-acc 0.62 come from?

https://github.com/ContinualAI/reproducible-continual-learning/blob/eafcb218c08d6e6234799d0171b4a19bd6ac1d89/strategies/target_results.csv#L18

I also checked the original papaer. The last exp top1-test-acc is about 0.49 (amazing!)

I tried my own implementation code and the avalanche library. I can't reach such high results. Could u pls give some insights? I'm so confused right now.

Hope to get ur early replay. Thx.

[AndreaCossu]

Hi @qsunyuan , can you please open a new issue? @rudysemola maybe you can help answering?

[qsunyuan]

Thx for ur quick replay. I will open a new issue.

[rudysemola]

Hi @qsunyuan.
If I remember well, the last result in the plot shown to you is the accuracy achieved for the last experience (0.49).
The metric used here is the average incremental accuracy defined by the authors in Section 4. (Experiments, Benchmark protocol part).
If you see the image, taken from the paper (Table 1 a), the result achieved in the paper achieved is 0.641 for 10 classes using this definition of the metric, not 0.49.
0.62 was achieved using the same definition in the paper.

[rudysemola]

If you want another similar code to reproduce the results in the paper (near to 0.641) you can use this old code, is basically the same but you never know maybe can work and help you ;)

https://github.com/rudysemola/reproducible-continual-learning/blob/024d26585dc9b3917d8827be376b57dd3b1eaa40/strategies/iCARL/experiment.py

[qsunyuan]

Thx for ur help. It really helps me a lot. I will try the codes right now.
Thx again. Best wishes. Have a good day.

[qsunyuan]

I tried ur link @rudysemola and this one https://github.com/ContinualAI/avalanche/blob/master/examples/icarl.py

I achieved a result of about 0.62.

Unfortunately I also did not achieve the result of 0.64.

[AndreaCossu]

Hi :) Minor changes in the performance with respect to the original paper are often due to slightly different training modalities (e.g. learning rate scheduler) which are not always easy to investigate or are not disclose in the paper. Therefore, as a policy, we allow for a 2% slack in accuracy during tests. Our target result for iCarl is 0.62. If you manage to close this gap, please let us know!