[Feature Request] lazy implementation of priority updates for prototype `ReplayBuffer`

[paulomarciano]

Motivation

The modular prototype replay buffer includes a feature that marks changes in the storage for update in the different components. This is especially useful for prioritized sampling in the PrioritizedSampler, in which we use this to update the priority of sampling each element. Currently, this update is done synchronously with updates to storage, but we could do better by storing the updated indexes and only updating the priorities when we need them.

Solution

Something like

PrioritizedSampler:
# ...
    def mark_update(self, index):
        self._updates.extend(index)

    def sample(...):
        if self._updates:
            self.update_all()
        # sample continues

Alternatives

We could also have the update running as a background task on a timer.

Checklist

• I have checked that there is no similar issue in the repo (required)