ScriptModule from Stream / ByteArray

[oliver-bernhardt]

Hey

I was wondering whether there are any means to allow model loading from Stream (instead of only from file) for ScriptModule.cs?
This would be very important in cases where we have propitiatory model architecture that we do not want to store in a human readable form on the clients hard-drive.

Have a nice day

[NiklasGustafsson]

The ScriptModule is loaded by native code, so I don't believe it could be made to work from a .NET stream. Just let me ask you this -- where is the TorchScript module coming from in the first place? How do you get it into a stream without going via a file?

[NiklasGustafsson]

There is a version of the native JIT load() that looks like this:

TORCH_API Module load(
    std::istream& in,
    c10::optional<c10::Device> device = c10::nullopt,
    bool load_debug_files = true);

TORCH_API Module load(
    std::istream& in,
    c10::optional<c10::Device> device,
    ExtraFilesMap& extra_files,
    bool load_debug_files = true);

So, if a .NET MemoryStream (or rather, it's raw contents) could be marshaled into an instance of a C++ std::istream, then it should theoretically work.

There is also:

TORCH_API Module parse_and_initialize_jit_module(
    std::shared_ptr<char> data,
    size_t size,
    ExtraFilesMap& extra_files,
    c10::optional<at::Device> device);

but I haven't found any documentation on that function.

[NiklasGustafsson]

If you think this is important enough, please turn it into an issue for tracking.

[NiklasGustafsson]

Seems like it should be doable to construct an std::istream from a byte array, going via this interface:

https://en.cppreference.com/w/cpp/io/basic_streambuf

(the last many years, I've mostly done algorithmic work in C++, so my C++ I/O knowledge is somewhat rusty).

[NiklasGustafsson]

I'm trying various things, but so far, no luck. Surprising that there's no standard way to get a stream from a byte array in C++. There's something in Boost, but I don't want to take a dependency on that if it can be avoided.

[NiklasGustafsson]

Actually, this works, but supposedly (according to StackOverflow) makes a copy, or maybe even two:

std::istringstream stream(std::string(bytes, size));
auto res = torch::jit::load(stream, torch::Device(dev, index));

With that in the native code, I can do this in .NET:

var bytes = File.ReadAllBytes(@"func.script.dat");
using var m = torch.jit.load<Tensor, Tensor, Tensor>(bytes);

[oliver-bernhardt]

Again, one does not need to overcomplicate it. It's completely sufficient also to just pass it as a byte[]. Those should not be an issue to pass between .NET and C++ right?

…

-Biognosys Confidential-

________________________________ From: Niklas Gustafsson ***@***.***> Sent: Friday, September 22, 2023 10:58 PM To: dotnet/TorchSharp ***@***.***> Cc: Bernhardt, Oliver ***@***.***>; Author ***@***.***> Subject: Re: [dotnet/TorchSharp] ScriptModule from Stream / ByteArray (Discussion #1094) **EXTERNAL EMAIL** Actually, this works, but supposedly (according to StackOverflow) makes a copy: std::istringstream stream(std::string(bytes, size)); auto res = torch::jit::load(stream, torch::Device(dev, index)); — Reply to this email directly, view it on GitHub<#1094 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANWHQ2BADNWSYWBQFGWL3JLX3X3YRANCNFSM6AAAAAA477F5YY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[NiklasGustafsson]

Well, yes, that's what I'm doing on the .NET side in this experiment. What's complicated is on the native side, where I have to get the byte array into an std::istream, preferably without copying.

[NiklasGustafsson]

@oliver-bernhardt, if you are willing to help test the solution, I can push to an off-main branch and you can test it with a specific model that you have in mind. That would require building from source, though.

[oliver-bernhardt]

Hi,
we would for sure be happy to test this. We are just not quite there yet.
At the moment we were looking into it from a feasibility point of view but the model loading was a no-go.

Please provide me with a branch on which I can test this on :)

[NiklasGustafsson]

Okay, so there's a branch 'jitload' with the changes. You'll have to build from source in order to test.

Right now, the API overloads torch.jit.load() to take a byte array in addition to the file name version. I did not add an overload for torch.jit.save() yet -- I want to make sure this meets your needs first.

[phineasng]

Sorry for the late reply. This seems to be working for us (cc @oliver-bernhardt). Would it be possible to add to merge, @NiklasGustafsson ? and when do you think it would be possible to have a minor release with this feature? Thanks a lot in advance!

[NiklasGustafsson]

I can take a look later today or tomorrow. It's diverged enough that the merge will have to be manual, which is okay, it's not a huge edit.

[NiklasGustafsson]

This will be in the next release.

[phineasng]

awesome, thanks a lot!