Not working using cuda

[Fable67]

Variables self.slow_weights are always on cpu.
You can easily fix this by adding a .to() method in Ranger class like so:

def to(self, device):    
    if device is "cuda":
        for i in range(len(self.slow_weights)):
            for j, w in enumerate(self.slow_weights[i]):
                self.slow_weights[i][j] = w.cuda()
    elif device is "cpu":
        for i in range(len(self.slow_weights)):
            for j, w in enumerate(self.slow_weights[i]):
                self.slow_weights[i][j] = w.cpu()

[pabloppp]

Are you by any chance creating your Ranger optimizer before moving your model to CUDA?

If so, what you're experiencing is a know error on optimizers that initialize internal state based on the model parameters, like Adagrad, and would be solved just by instantiating your optimizer after you move your model to CUDA.

Here's a post where they mention this issue: https://discuss.pytorch.org/t/effect-of-calling-model-cuda-after-constructing-an-optimizer/15165/7

It's not so common moving the optimizer parameters to cpu/cuda after instantiating them, so I think adding a 'to' method to the optimizer with that purpose, as you suggest, would not be very familiar to regular PyTorch users.

[lessw2020]

Thanks everyone for the feedback on both sides here.
@pabloppp is correct, that PyTorch expects the optimizer to be instantiated after the model is created.
That ordering ensures the correct GPU handling.