Saving torchscript model in C++

[oplatek]

🚀 Feature

Allow torch::jit::script::Module in C++ to serialize the model back to disc.

I submitted the issue based on the thread https://discuss.pytorch.org/t/how-to-save-torchscript-model-using-cpp/59743/10

Motivation

General motivation:

• serialize model to disc allows two-way interchangeability of environments between Python and C++ (loading the models is supported in C++)
• two-way interchangeability between C++ and Python significantly speeds up debugging of the production C++ code

My particular use case:

1. I train seq2seq model in Pytorch
2. I save the jit traced model for a single step of decoding seq2seq to disc
3. I load the model in C++ Libtorch, run viterbi/beam-search decoding implemented in C++ using the single-step forward method as a single step of decoding.
4. I fine-tune the C++ hyperparameters/ production setup
5. I want to serialize the torchscript model in C++ from the production (together with my hyperparameters)
6. The bundle of saved torchscript model and the hyperparameters are shipped to production

Pitch

Please implement C++ version of https://pytorch.org/docs/stable/jit.html#torch.jit.save

Alternatives

• Workaround for my particular use case is to keep in the memory the binary data loaded from the model directly "serialized" from python PyTorch code.
  • Note: this workaround fails if we need to adapt the deployed model using a little enrolment data in our C++ application - our plan. For that, we would need to switch back to Python environment which is not possible (imagine mobile phones/ servers)
• I am interested in other workarounds ... feel free to suggest them

cc @suo

[oplatek]

Related - serialization of tensors between C++ and Python: #20356 (comment)

[eellison]

@driazati this should be possible right ?

[driazati]

Yeah we already have the ability to save to a std::ostream in C++

https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/api/module.h#L208

@oplatek does this API do what you want or are you looking for something else?

[oplatek]

@driazati this is it! I do not know how I missed it.

[driazati]

Cool, closing this issue then. Feel free to reopen if you run into more problems.

[fkxie]

@oplatek Hi, would you please tell me how did you solve the problem?

Thanks in advance!

[oplatek]

I used the API as suggested by @driazati. https://github.com/pytorch/pytorch/blob/3ab88c39038b75ff22f9985b415f7af943c6303b/torch/csrc/jit/api/module.h#L214

On Mon, 5 Oct 2020 at 10:49, fkxie ***@***.***> wrote: @oplatek <https://github.com/oplatek> Hi, would you please tell me how did you solve the problem? Thanks in advance! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABX7EWGSPVONTOYCLJFWJDSJGB7BANCNFSM4LUCPPMA> .

-- Ondřej Plátek

[fkxie]

Yeah，I see. Thanks for reply！ And I’m pleasure to know how to use it!-- 发自新浪邮箱客户端 在 10月6日 00:55，Ondrej Platek <notifications@github.com> 写道： I used the API as suggested by @driazati. https://github.com/pytorch/pytorch/blob/3ab88c39038b75ff22f9985b415f7af943c6303b/torch/csrc/jit/api/module.h#L214

…

On Mon, 5 Oct 2020 at 10:49, fkxie ***@***.***> wrote: @oplatek <https://github.com/oplatek> Hi, would you please tell me how did you solve the problem? Thanks in advance! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABX7EWGSPVONTOYCLJFWJDSJGB7BANCNFSM4LUCPPMA> .

-- Ondřej Plátek &mdash; You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[maxfiedler]

I am bit confused:
in Dec 2019 yf225 wrote that saving a model written in the C++ front end to TorchScript Module is not possible: https://discuss.pytorch.org/t/how-to-save-torchscript-model-using-cpp/59743/10?u=mxf

Is this a way to do this now? Or am I misunderstanding the use-case that you are pursuing here?

[maxfiedler]

and what is the purpose of the _save_for_mobile functions in https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/api/module.h ?

[Stefan-1313]

Yeah we already have the ability to save to a std::ostream in C++

https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/api/module.h#L208

@oplatek does this API do what you want or are you looking for something else?

I'm trying to do the same thing, but using the save using std::ostream does not work for me. I get a corrupted file which I cannot load (The load using std::istream even gives an exception).

Can you provide me with an example? Thanks, I would be very grateful!

[tom-huntington]

I am bit confused: in Dec 2019 yf225 wrote that saving a model written in the C++ front end to TorchScript Module is not possible: https://discuss.pytorch.org/t/how-to-save-torchscript-model-using-cpp/59743/10?u=mxf

Is this a way to do this now? Or am I misunderstanding the use-case that you are pursuing here?

There is no way to take a model in c++ that inherits from torch::nn::Module and trace it to create a torch::jit::Module.

But you can load a torch::jit::Module and then save it again in c++,

torch::jit::Module module = torch::jit::load("mlp.pt");
module.save("cppSavedJitModule.pt");
module = torch::jit::load("cppSavedJitModule.pt");

Edit: If you really need to export your c++ model to onnx, you can actually create a torch::jit::Module by hand, using torch::jit::tracer::trace().
See https://github.com/hasktorch/hasktorch/blob/a752f4c868caf386aeefafec9ab97e5810728f50/libtorch-ffi/src/Torch/Internal/Unmanaged/Type/Module.hs#L271
and https://github.com/pytorch/pytorch/blob/master/test/cpp/jit/test_autodiff.cpp#L152
You have to:

for (auto& p : model->parameters())
    {
        p.set_requires_grad(false);
    }

so you cant train the resulting jit module, but the inference works in c++, I didn't actually try converting to onnx though. Also I wasn't able to vary the batch size in the resulting jit module. Control flow probably wont work too