Speed difference between two algorithm 'blocks' and 'lstm'

[toco2270853]

Hello, thanks for the good code.
I find that while running CIFAR10 Experiments, there is about 40x speed difference between 'blocks' and 'lstm'. Is it normal?

[sarthmit]

Thanks for your interest in the paper!

I am not sure what is the margin for speed decrease but RIMs and BRIMs are in general considerably slower than LSTM. This is because of the following issues -

1. We do a sparsification of the RNN matrices after each gradient update to make sure that the modules are independent
   

   BRIMs/CIFAR10/rnn_models.py

   Line 73 in f8af67e

   self.model.blockify_params()

2. There is a top-k selection criteria that is quite a bottleneck in speed
   

   BRIMs/CIFAR10/blocks_core_all.py

   Line 136 in f8af67e

   bottomk_indices = torch.topk(iatt[:,:,0], dim=1,

I do encourage to check the following implementation of RIMs out (https://github.com/dido1998/Recurrent-Independent-Mechanisms) that attempts at speeding up RIM computations by solving the issue illustrated in (1.) by using groupRNN.