Wrong behaviour when test_start_index=0

[expectopatronum]

When test_start_index=0 the first 10 test samples are used and not the first one per class. This is caused by checking if test_false_index is True. This can be fixed by using None when no index should be used. I fixed it here and can make a pull request if you agree that this is the intended behaviour.

Best regards
Verena

[nimarb]

Hi Verena, thanks a ton for bringing this to my attention.

Currently, the default value of test_start_index is 0, as can be seen in utils.py -> get_default_config().

Unfortunately, I can't replicate this error. For example for CIFAR10, with the default values I get an array of samples to be tested: [3, 6, 25, 0, 22, 12, 4, 13, 1, 11]. Here, CIFAR10.dataset[3] is the first occurrence of the class with the label of 0, [6] the first one of class with label 1 and so on. The first sample in the test dataset is of class 3 which is found correctly to have the index 0.

The intended behaviour is, that test_start_index is the number of the first image per class in the test set for which all influences are calculated. Then, you can control with test_sample_num for how many test samples per class the influence is calculated for. Ideally, test_sample_num should've been named test_sample_num_per_class and test_start_index should've been named test_start_index_per_class. Setting test_sample_num = 1 and the test set having 10 classes will result in influences being calculated for test[0] ... test[9], as it's being calculated for one sample of each class. This behaviour is actually quite inflexible and I have to document that better & implement an easy way to enable calculate for just a single test sample.

Does this make sense or did I misunderstand your bug report perhaps?
Thanks, Nimar

[expectopatronum]

Hi Nimar!
Thanks for your response! Yes, it does make sense and that's what I thought. It's weird that you can't reproduce it, I'm sorry if I made a mistake. I am currently working towards a deadline, after that I'll try to reproduce it.

Best regards
Verena

[nimarb]

Ok great, let me know. By the way, which dataset are you using?

[expectopatronum]

Hi nimarb,
I tried really hard but I can't reproduce the error :) Since you mentioned the dataset, we had some class dervie from the CIFAR dataset that you also use and I think that we had fix something in there, so this might have been the cause. (It is still weird that I fixed this issue / or thought I did by changing False to None).
Sorry for causing any inconvenience, that's probably what happens when I am under the pressure of a deadline :)

Best regards
Verena