Moving this issue from dotnet/machinelearning#6492 since the work required would need to happen in TorchSharp.