Passing optimizer state dictionary from PyTorch to TorchSharp.

[TheBlackPlague]

Is it possible to pass the optimizer state dictionary from PyTorch to TorchSharp?

I know it's to do so with the model state dictionary (using the script provided to save the model state dictionary in the TorchSharp format), but that method wouldn't work for the optimizer state dictionary.

Let me know.

[NiklasGustafsson]

The TorchSharp optimizers should all have:

public void save_state_dict(string location)
public void load_state_dict(string location)
public void save_state_dict(System.IO.BinaryWriter writer)
public void load_state_dict(System.IO.BinaryReader reader)

Note, however, that in some version changes, the internal state of optimizers may change, and therefore there is no guarantee that optimizer state will work across version updates.

[NiklasGustafsson]

Sorry -- you were asking about saving state in PyTorch and loading it in TorchSharp. No, no effort has been made to do so. We'd have to do the same thing we did for exporting/importing model state, if that is a significant and common use case.

[TheBlackPlague]

Well, it is a pretty significant and common use to set everything up in PyTorch, and then transfer the pipeline to something more safe, robust, and performant like TorchSharp (mainly because of the language).

[NiklasGustafsson]

Okay, just so I understand -- you want to start training in PyTorch and continue it in TorchSharp?

[TheBlackPlague]

Okay, just so I understand -- you want to start training in PyTorch and continue it in TorchSharp?

Yes.

[GeorgeS2019]

@NiklasGustafsson

something more safe, robust, and performant like TorchSharp (mainly because of the language).

I wonder what else are the main use cases to transfer the pytorch pipeline to TorchSharp, if possible, in terms of popularity

• optimizer state dictionary from PyTorch to TorchSharp
• Torchscript for deployment from PyTorch to TorchSharp
• PyTorch to ML.NET by passing TorchSharp using PyTorch Onnx Export

In other words, TorchSharp as the downstream pipeline for deployment after all training, validation, etc have been done in PyTorch.

@TheBlackPlague
Thanks for pointing out this potentially a main use case for deployment using TorchSharp

[NiklasGustafsson]

Okay, just so I understand -- you want to start training in PyTorch and continue it in TorchSharp?

Yes.

Alright. This will take a while.

[NiklasGustafsson]

Do you also need to go back? I mean, save optimizer state in TorchSharp and load it in PyTorch?

[TheBlackPlague]

I shouldn't need to. But I'm sure someone will find a case where that will be of use to them. It's always good to have more than less.

[NiklasGustafsson]

I'm working on an implementation using the same binary format that TorchSharp uses for internal purposes. There are a few changes I'm making to that, so optimizer state will not carry forward bettween TorchSharp versions. In other words, it will be a breaking change from that perspective.

[TheBlackPlague]

Wouldn't it be worth creating a legacy carry-over method for those that may need it?

[NiklasGustafsson]

Maybe if we were at v1.0 or higher. Until we have some sort of consensus that TorchSharp was product-quality, we could move to that, but there are a lot of things that aren't in place, like documentation, more sophisticated examples, and users are still finding things missing, so we're not there yet.

[GeorgeS2019]

@NiklasGustafsson

Maybe if we were at v1.0 or higher.

there are a lot of things that aren't in place, like documentation, more sophisticated examples, and users are still finding things missing

6 months ago, I would say we need another 2 years.

From the speed NEW HELP/SUPPORT, AND INTEREST coming to this project, v1.0 is reachable within the next 6 months OR even shorter.

[NiklasGustafsson]

See this wiki page for instructions on how to transfer optimizer state from PyTorch to TorchSharp (once the PR goes through and we make the next release):

https://github.com/dotnet/TorchSharp/wiki/Sharing-Model-Data-between-PyTorch-and-TorchSharp