Inquiry about Pre-trained Model & Parameter Setup

[longbai1006]

Many thanks for this wonderful framework! It really helps our work a lot!

I have some questions about your experiment setup.

1. I have reproduced your 10-stage CIFAR-100 experiments on my own PC (3*3090). The results are as followed:

I followed all the bash files and parameters you have set up, but the results seem to be much lower than yours. Is that because you use an ImageNet pre-trained ResNet? Thanks!

2. I found you set different model optimization parameters (e.g. learning rate, epoch, milestone, etc.) for each continual learning approach (instead of setting the same hyperparameter policy for all the continual learning approaches). I was wondering whether this kind of parameter setup could be considered a "fair" comparison?

Thanks in advance for your answer!

[G-U-N]

Thanks for your interest.

For the first problem, I guess you might confuse the average accuracy with the last session's accuracy. We report the average accuracy of all sessions, which is also the benchmark in most class-incremental learning papers, while your table seems to be the accuracy of the last stage.

For the second problem, different methods have different suitable parameters. For example, lwf and ewc may need smaller weight decay while icarl requires a larger one. We set slightly different parameters between different methods to fully reflect the performance of these methods.

[longbai1006]

Thanks for your answer!

Could I confirm my understanding again for the first question because it does differ from another continual learning library I am working on : )

For my first question, is that means:
On task 0, we get accuracy0 on class 0-9;
on task 1, we get accuracy1 on class 0-19;
on task 2, we get accuracy2 on class 0-29;
...
on task 9, we get accuracy9 on class 0-99;
then we get the average accuracy of (accuracy0, accuracy1 ... accuracy9)?

[G-U-N]

You're right.

[longbai1006]

Got it. Thanks.