RetrievalPolicy and StoragePolicy for Rehearsal methods

[Mattdl]

I was thinking to implement CoPE in the framework. But there remain two main TODOs for any online data incremental, or replay methods in general.

First, it still seems like the data incremental learning issue remains unsolved (#360), so it'll be hard to reproduce results to check if the implementation is working. That is, we should be able to define a data stream still in task-like streams, whereas processing can happen per batch. This is often used to enable comparison to non-data incremental methods (which need a task-like datastream).

Second, Also there are a lot of replay based methods possible, and I think we should consider using generic objects (e.g. StoragePolicy, RetrievalPolicy) so that we can easily port these for other (replay)methods.

At this point we have a ReplayPlugin, but if we could make this more specific by passing a Retrieval and StoragePolicy to the ReplayPlugin. I see the 'ReplayPlugin' as the plugin actually using the memory (e.g. for optimization), so methods like CoPE could actually be implemented similarly.
The Retrieval and StoragePolicy are defining how to sample from the data stream, and how to sample from the buffer respectively. Replay (ER) or CoPE are actually independent of which retrieval/storage strategy is used.

Furthermore, this modular approach also allows us to easily combine methods, such as GSS (which is a StoragePolicy based on the gradients) or MIR (which is a retrieval strategy based on the loss), which we could then just plug in for ER, CoPE or any other replay based method (iCaRL,...).

We should probably break this down into multiple sub-issues:

1. data incremental (already existing: Support for data incremental scenario #360)
2. RetrievalPolicy: Random, Class-balanced random,...
3. StoragePolicy: Reservoir, Class-balanced reservoir, herding (iCaRL),...
4. Adapting the existing replay methods with a retrieval/storage policy

[AntonioCarta]

I agree with separation of retrieval and storage. It seems to me that the storage policy is the most complex one, and it will probably need to use the full plugin API because it needs access to the strategy's internal state, especially for complex policies (loss, mini-batch, ...). Retrieval policies may be implemented as custom pytorch DataLoaders. For example, we already have some dataloaders that create minibatches by combining data from the current experience and the replay buffer.

I'm not sure if DataLoaders are enough to implement all the storage policies. @Mattdl what do you think? For example, some replay strategies combine current experiences with the buffer to create mini-batches (especially for class-incremental scenarios), while others use only the buffer (I think this is used in data incremental scenarios). Basically, in the first case you have new exp -> model update (using buffer+exp) -> buffer update, while in the second you have new exp -> buffer update -> model update (using buffer). Maybe it's easier to ignore the first case, leave the current ReplayPlugin as is, and implement your proposed parametric replay which works like the second scenario.

In the end, I would expect to use replay like this:

replay = ClassBalancedReplay(retrieval_policy='class_balanced_dataloader', ...)

or alternatively:

storage_policy = RandomBuffer()
replay = Replay(storage_policy=RandomBuffer() , retrieval_policy='class_balanced_dataloader', ...)

@Mattdl you probably have much more experience than me with replay strategies. Let me know if I'm missing anything.
Can you start with an implementation of the generic API using a random retrieval and reservoir? We will implement the more complex ones later as separate issues. I think once we have the basic API it should be easy. Keep in mind that AvalancheDataset now supports concatenation with + and subsampling with random_split, therefore random policies should be relatively straightforward to implement.

[Mattdl]

I favor the alternative you proposed to use the basic Replay plugin and then pass a storage_policy and retrieval_policy.
However, both for consistency and to allow more advanced retrieval policies (e.g. based on the loss etc.) I would make both of them Plugins, as in :

replay = Replay(storage_policy=RandomBuffer() , retrieval_policy=SomeRetrPolicy(), ...)

This way, other Replay-based plugins can just do something similar:

replay = GEM(storage_policy=PerExperienceBuffer() , retrieval_policy=RetrieveAll(), ...)

I will implement the initial easy ones and try to use them for my CoPE implementation.
Thanks for the help!

[AntonioCarta]

If you need any comments on a preliminary implementation you can open a PR and tag it with [WIP] until it's ready to merge.

[AntonioCarta]

I'm closing this for now. We can open separate issues for the missing policies in the future.