Autotune p_keep parameter for RISE

[cwmeijer]

We think that std of the probability of the predicted correct class on all masked instances of an image should be somewhere between 0.2 and 0.4. Under 0.1 we didn't see any good looking heatmaps. Between 0.1 and 0.2 we often saw some artifacts and strange blobs in the heatmaps. From 0.2 onwards we didn't see such randomish errors anymore. An std of 0.5 is possible in theory and will probably also be fine in terms of the resulting heatmap. Conclusion: at least std of 0.2.

Proposed algorithm:

• use small number of masks for speed (let's say 50)
• repeat:
  • try p_keep value and calculate std
• take p_keep with high enough (highest?) std and use this for a computation with enough masks for quality (2000+?)

[cwmeijer]

We need to chose some value for n_masks when doing sample runs to determine the p_keep. Nb. this is not the actual n_masks that is used for the real run, so after determining p_keep.
I did a bincount for these bins:
[ 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
For a couple of n_masks values I ran the p_keep autotuning algorithm 100 times.

For images:

When using 20 n_masks we got this p_keep bincount: [ 1 7 19 24 21 18 8 2]
When using 30 n_masks we got this p_keep bincount: [ 0 4 11 30 23 23 9]
When using 50 n_masks we got this p_keep bincount: [ 0 3 16 35 28 14 4]
When using 100 n_masks we got this p_keep bincount: [ 0 3 14 37 23 21 2]
When using 200 n_masks we got this p_keep bincount: [ 0 0 16 37 32 15]

It seems 20 is not enough to have a good chance of getting a good p_keep as there is a l lot of variation in the chosen p_keep. For 200 every sample returns a
reasonable p_keep but is a bit slow to be practicle. I think we should use 100 to be on the save side.

For text:

When using 20 n_masks we got this p_keep bincount: [ 0 0 2 11 37 41 7 2]
When using 30 n_masks we got this p_keep bincount: [ 0 0 1 9 33 42 14 1]
When using 50 n_masks we got this p_keep bincount: [ 0 0 0 5 50 42 3]
When using 100 n_masks we got this p_keep bincount: [ 0 0 0 7 61 31 1]

It seems 20 is not enough to have a good chance of getting a good p_keep. It seems 50 n_masks is enough here.
To be on the safe side we go for 100 anyway.