How to use model.pt in GPU whith C++ API

[joy-yjl]

I wrapped a pt with Torch Script, and load it with c++ api.

I want to use it in GPU, I can do model.cuda() in python . How can I do it in C++.

[bhack]

See #12686 (comment).
Also I would know if the network weights could be loaded after torch::jit::load().
/cc @goldsborough

[goldsborough]

@joy-yjl You can write model->to(at::kCUDA). Does that work for you?

@bhack you mean whether you can first load the model, and then the weights after? Please elaborate.

[bhack]

@goldsborough Yes, I meant to load it with torch::jit::load() and then load or change some trained weights also multiple time directly in c++.
I.e So that we can trace with the model definition and just initialized weights and than change the weights with a trained ones as we need without tracing again.

[goldsborough]

@bhack You can always access the parameters after and make changes. E.g. use module->get_parameter(name). Generally see https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/script/module.h for things you can do with a module.

[bhack]

Yes I saw the header for single parameter but I meant something like the the equivalent of the_model.load_state_dict(torch.load(PATH))

[joy-yjl]

model->to(at::kCUDA)
It is work for me .thanks!

[bhack]

@goldsborough I see only a get_parameters at https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/script/module.h#L368 but not a method to set again parameters at once.

[lenlen]

@goldsborough I need also this feature to set all parameters at once to avoid a loop for each parameters, to load the weights in my traced network.

[bhack]

@lenlen If you are interested I've opened a specific ticket at #13383

[Joshua-Cardio]

model->to(at::kCUDA)
It is work for me .thanks!

Hi Joy-yil,

Could you share a little more about your setting?

I follow the tutorials on https://pytorch.org/tutorials/advanced/cpp_export.html, and add

model->to(at::kCUDA)

But errors is thrown when execute the model
I auto output = module->forward(inputs).toTensor();

Error:
what(): Expected object of backend CPU but got backend CUDA for argument #2 'weight'

Thanks,

[goldsborough]

@Joshua-Cardio did you move your inputs to CUDA? e.g. with input.to(at::kCUDA) before passing it to the model

[Joshua-Cardio]

Thanks @goldsborough.

when using
std::vectortorch::jit::IValue inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));
inputs.to(at::kCUDA);
an error occurs,
'class std::vector<c10:IValue, std::allocatorc10::IValue>' has no member named 'to'

Later I follow your guidance in #12506, and
add tensor_image.to(at::kCUDA). It worked. Thanks!

[bhack]

Yes #12506 it really needs to be included in the tutorial.

[goldsborough]

I'll put it on my list of things to do to expand the tutorial around GPU usage and conversion from and to tensor types :)

[bhack]

@goldsborough I think that the tutorial could explain also torch.no_grad in c++ and python interfaces. I think that it is a common practice with traced models but it is undocumented.

[ananth-jauntxr]

Any update on GPU usage? :)

[soumith]

@ananth-jauntxr it's in the 2nd comment of the thread

[cindycia]

Thanks @goldsborough.

when using
std::vectortorch::jit::IValue inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));
inputs.to(at::kCUDA);
an error occurs,
'class std::vector<c10:IValue, std::allocatorc10::IValue>' has no member named 'to'

Later I follow your guidance in #12506, and
add tensor_image.to(at::kCUDA). It worked. Thanks!

I got the same issue.

Can you share with me your final solution in more details?

[Eniac-Xie]

Hi, @soumith @goldsborough how can I set gpu id in C++ code? It seems that cudaSetDevice() didn't work.
Thank you!

---update
according to: https://pytorch.org/cppdocs/notes/tensor_creation.html
torch::Tensor gpu_two_tensor = float_tensor.to(torch::Device(torch::kCUDA, 1));