Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use model.pt in GPU whith C++ API #13245

Closed
joy-yjl opened this issue Oct 29, 2018 · 19 comments

Comments

Projects
None yet
10 participants
@joy-yjl
Copy link

commented Oct 29, 2018

I wrapped a pt with Torch Script, and load it with c++ api.

I want to use it in GPU, I can do model.cuda() in python . How can I do it in C++.

@joy-yjl joy-yjl changed the title How to use model.pt in GPU How to use model.pt in GPU whith C++ API Oct 29, 2018

@bhack

This comment has been minimized.

Copy link

commented Oct 29, 2018

See #12686 (comment).
Also I would know if the network weights could be loaded after torch::jit::load().
/cc @goldsborough

@zou3519 zou3519 added the module: cpp label Oct 29, 2018

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Oct 29, 2018

@joy-yjl You can write model->to(at::kCUDA). Does that work for you?

@bhack you mean whether you can first load the model, and then the weights after? Please elaborate.

@bhack

This comment has been minimized.

Copy link

commented Oct 29, 2018

@goldsborough Yes, I meant to load it with torch::jit::load() and then load or change some trained weights also multiple time directly in c++.
I.e So that we can trace with the model definition and just initialized weights and than change the weights with a trained ones as we need without tracing again.

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Oct 29, 2018

@bhack You can always access the parameters after and make changes. E.g. use module->get_parameter(name). Generally see https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/script/module.h for things you can do with a module.

@bhack

This comment has been minimized.

Copy link

commented Oct 29, 2018

Yes I saw the header for single parameter but I meant something like the the equivalent of the_model.load_state_dict(torch.load(PATH))

@joy-yjl

This comment has been minimized.

Copy link
Author

commented Oct 30, 2018

model->to(at::kCUDA)
It is work for me .thanks!

@joy-yjl joy-yjl closed this Oct 30, 2018

@bhack

This comment has been minimized.

Copy link

commented Oct 30, 2018

@goldsborough I see only a get_parameters at https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/script/module.h#L368 but not a method to set again parameters at once.

@lenlen

This comment has been minimized.

Copy link

commented Oct 30, 2018

@goldsborough I need also this feature to set all parameters at once to avoid a loop for each parameters, to load the weights in my traced network.

@bhack

This comment has been minimized.

Copy link

commented Oct 31, 2018

@lenlen If you are interested I've opened a specific ticket at #13383

@Joshua-Cardio

This comment has been minimized.

Copy link

commented Nov 1, 2018

model->to(at::kCUDA)
It is work for me .thanks!

Hi Joy-yil,

Could you share a little more about your setting?

I follow the tutorials on https://pytorch.org/tutorials/advanced/cpp_export.html, and add

model->to(at::kCUDA)

But errors is thrown when execute the model
I auto output = module->forward(inputs).toTensor();

Error:
what(): Expected object of backend CPU but got backend CUDA for argument #2 'weight'

Thanks,

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2018

@Joshua-Cardio did you move your inputs to CUDA? e.g. with input.to(at::kCUDA) before passing it to the model

@Joshua-Cardio

This comment has been minimized.

Copy link

commented Nov 1, 2018

Thanks @goldsborough.

when using
std::vectortorch::jit::IValue inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));
inputs.to(at::kCUDA);
an error occurs,
'class std::vector<c10:IValue, std::allocatorc10::IValue>' has no member named 'to'

Later I follow your guidance in #12506, and
add tensor_image.to(at::kCUDA). It worked. Thanks!

@bhack

This comment has been minimized.

Copy link

commented Nov 1, 2018

Yes #12506 it really needs to be included in the tutorial.

@goldsborough

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2018

I'll put it on my list of things to do to expand the tutorial around GPU usage and conversion from and to tensor types :)

@bhack

This comment has been minimized.

Copy link

commented Nov 6, 2018

@goldsborough I think that the tutorial could explain also torch.no_grad in c++ and python interfaces. I think that it is a common practice with traced models but it is undocumented.

@ananth-jauntxr

This comment has been minimized.

Copy link

commented Dec 11, 2018

Any update on GPU usage? :)

@soumith

This comment has been minimized.

Copy link
Member

commented Dec 12, 2018

@ananth-jauntxr it's in the 2nd comment of the thread

@cindycia

This comment has been minimized.

Copy link

commented Dec 13, 2018

Thanks @goldsborough.

when using
std::vectortorch::jit::IValue inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));
inputs.to(at::kCUDA);
an error occurs,
'class std::vector<c10:IValue, std::allocatorc10::IValue>' has no member named 'to'

Later I follow your guidance in #12506, and
add tensor_image.to(at::kCUDA). It worked. Thanks!

I got the same issue.

Can you share with me your final solution in more details?

@Eniac-Xie

This comment has been minimized.

Copy link

commented Jan 21, 2019

Hi, @soumith @goldsborough how can I set gpu id in C++ code? It seems that cudaSetDevice() didn't work.
Thank you!

---update
according to: https://pytorch.org/cppdocs/notes/tensor_creation.html
torch::Tensor gpu_two_tensor = float_tensor.to(torch::Device(torch::kCUDA, 1));

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.