Cnn influence example

[Xuzzo]

Description

This PR introduces an example where influence functions are used for CNNs.

Changes

• Conjugate gradient calculated implicitly through scipy LinearOperator. Closes Reduce memory usage in influence calculation #128
• IF computation now supports multi-dimensional input. Closes IF calculation breaks with multi-dimensional input #209
• New notebook with examples and theory
• Progress on Investigate flakiness of influence values #130
• Closes Add notebooks requirements file #221

Checklist

• Wrote Unit tests (if necessary)
• Updated Documentation (if necessary)
• Updated Changelog
• If notebooks were added/changed, added boilerplate cells are tagged with "nbsphinx":"hidden"

[Xuzzo]

• Removed caching of results. Kept the one for the model
• I have added the difference in score between corrupted and simple scores in the last table
• Fixed tqdm for model training and influence calculation

[AnesBenmerzoug]

@Xuzzo Thanks for your work! It looks better now.

The main thing I noticed is that in the imagenet notebook when you load the dataset it prints information that is useless to the notebook reader:

Perhaps we should call datasets.utils.logging.set_verbosity_error() in the load_preprocess_imagenet function.

[mdbenito]

I've looked a bit at the supporting code but couldn't check the notebooks yet. Will do asap

[Xuzzo]

MR comments should be addressed.

[AnesBenmerzoug]

I had another quick look at it. It looks good to me and can be merged!

[mdbenito]

Hi hi,

the supporting code is better now, thanks!

I haven't had time to check it in detail yet, but the notebook:

• contains several typos, please run it through a spell checker. Some of them are recurring and I have fixed many times in the past, like "data-points". 🙏🏽
• does not hide cells as instructed in the PR template and CONTRIBUTING.md (why?)
• Points to a whole book as reference for the main computation. At least one should mention the section or theorem.
• has a (useful) theory section which ignores the theory section in the review document draft, and is based on a paper which diverges from the usual theory of influence functions.

I understand that it makes sense to be consistent with the paper that is implemented, but a major selling point of the review and of a library like pydvl is that we provide a common framework and notation to understand any paper on influence functions. In particular, it is important to unify notation and be general enough. Koh2017 is a rather sloppy source because it fails to properly define the object that is being approximated and also diverges from the usual definitions. Custom definitions are ok, but we should strive to make the link to what is standard in the literature, I believe.

Now, the stub I wrote over a year ago is itself worse than sloppy, I'm not claiming it's any better (actually I just read it and it's pretty confusing, to put it mildly 😅). But it'd be nice to try to link both. It is ok to remain superficial in the notebook, if it is consistent with the review. Maybe you can start by mentioning the latter, and in the future work to improve that document (we should coordinate, though).

[Xuzzo]

To do, from discussion in #235:

• add notebook requirements
• check model saving
• check dataset ram usage: add comment on downsampling ds in notebook
• Change InternalDataset to TensorDataset
• fix pipeline