how to use cuda data in state (for train network)

[Alchemist77]

File "/home/jaeseok/Downloads/Reinforcement-Learning-Pytorch-Cartpole-master/rainbow/1-dqn/train.py", line 107, in
main()
File "/home/jaeseok/Downloads/Reinforcement-Learning-Pytorch-Cartpole-master/rainbow/1-dqn/train.py", line 89, in main
loss = QNet.train_model(online_net, target_net, optimizer, batch)
File "/home/jaeseok/Downloads/Reinforcement-Learning-Pytorch-Cartpole-master/rainbow/1-dqn/model.py", line 29, in train_model
states = torch.stack(batch.state)

RuntimeError: Expected object of backend CPU but got backend CUDA for sequence element 4 in sequence argument at position #1 'tensors'

---

Hello,
I have tested dqn in rainbow folder.

batch.state data is cuda data. I do not know how to change this line
states = torch.stack(batch.state)

I already did like,
states = torch.stack(batch.state).to(device)
states = torch.stack(batch.state).cuda()
It was not working. Do you have any suggestion

[g6ling]

Could you check your computer can use cuda?
torch.cuda.is_available()

[g6ling]

Maybe
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/model.py#L26-L30
All tensor is need to be added .to(device)

[Alchemist77]

Maybe
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/model.py#L26-L30
All tensor is need to be added .to(device)

Thank you g6ling.

I checked torch.cuda.is_available() and I can see cuda using print.
I added .to(device) to all tensor, but
states = torch.stack(batch.state)
This line has error.
If I remove lines that use cuda
online_net.train().to(device) ->online_net.train()
target_net.train().to(device)->target_net.train()
state = torch.Tensor(state).to(device)->state = torch.Tensor(state)
Then it works, but speepd is so slow becuase I do not use GPU..

[g6ling]

Did you change these parts?
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/train.py#L60
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/train.py#L69

After you changed this codes and code is not working, plz write your error.
Just not working I don't know what is problem.

[Alchemist77]

Did you change these parts?
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/train.py#L60
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/train.py#L69

After you changed this codes and code is not working, plz write your error.
Just not working I don't know what is problem.

I changed this line (removed .to(device) from original code)
state = torch.Tensor(state).to(device) -> state = torch.Tensor(state)
It means that i use only cpu for processing not gpu.

I still have this error when I run this line
states = torch.stack(batch.state).
RuntimeError: Expected object of backend CPU but got backend CUDA for sequence element 4 in sequence argument at position #1 'tensors'

I think that It means that batch.state data is processed on GPU (because you defined state with cuda in the code) but torch.stack is processed on CPU
Therefore, I am looking for how to use torch.stack with cuda, but I still did not find the way to work it.

[g6ling]

https://pytorch.org/docs/stable/torch.html#torch.tensor

device (torch.device, optional) – the desired device of returned tensor. Default: if None, uses the current device for the default tensor type (see torch.set_default_tensor_type()). device will be the CPU for CPU tensor types and the current CUDA device for CUDA tensor types.

Could you plz check your default device type? If default is cpu, you set device to cuda

[Alchemist77]

Dear g6ling,

I already set Cuda and I used another PyTorch code with Cuda.

I solved the problem.

https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/train.py#L69 ( next_state = torch.Tensor(next_state))
I add .to(device) because without .to(device), it is running on cpu and I always have erros described above (see my first question).

Then I have other errors when train_model is processed
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/model.py#L28 (actions = torch.Tensor(batch.action).float())
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/model.py#L29 (rewards = torch.Tensor(batch.reward))
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/model.py#L30 (masks = torch.Tensor(batch.mask))

Because action, reward, masks are running on CPU and it is used to calculate for target, so I added
actions = actions.cuda()
rewards = rewards.cuda()
masks = masks.cuda()
for running on GPU

Finally, I changed this line
https://github.com/g6ling/Reinforcement-Learning-Pytorch-Cartpole/blob/master/rainbow/1-dqn/model.py#L49 (return action.numpy()[0])
because I have to convert from GPU to CPU because I need to use numpy,
return action.cpu().numpy()[0]

I think that your implementation is good and a lot of help for me, but some people can have a different setup, so it can help someone who wants to use Cuda using PyTorch.

Thank you for your support.