Plan for Deep Learning in .NET #5918
Comments
You can take a look at how we exposed CNTK in TLC. Two styles, (1) fully manually designed NNs [do it myself], and (2) a simplification layer which created a network to your specifications [do it for me]. Both styles create custom NNs from scratch (no transfer learning). For 2, you specify the network type you want created, for example a fully-connected (aka feed-forward) NN, and specify the parameters of the network, like number of hidden layers, number of neurons wide, learning rate, training algo, etc. Then we create the NN for you, including your specific input/output sizes (which is a bit difficult to know in TLC / ML․NET before training). Exposing common neural network styles in a parameterized form makes it easy for a non-experienced user to create a functional NN, and also lets the AutoML do hyperparameter optimization easily on these networks (e.g. automatically alter the number of layers and learning rate in a fully-connected NN). |
|
Great roadmap, i am excited for this. |
|
Grate roadmap!! Simplifying the consumption of ONNX models is so necessary, I trained an ONNX model once but I could never understand the outpts. |
|
This is fantastic! Great to see the plan coming together :) |
|
The roadmap looks great! Specially not having to deal with C++ frameworks directly and being able to live in pure .NET workflows. |
You are avoiding dealing with C++, but now you are going to deal with python. Performance is going to be lost and the API convention of python will overwhelm .NET and C# developers. |
No Python is used in TorchSharp |
|
The roadmap should address the Model Explainability, built from bottom-up as a major milestone. I found the AutoMl.Net framework significantly improved the rapid modelling process, so I hope to see it as part of the roadmap. |
ONMX runtime is written in C++. I am also quite comfortable with C++, including being up to date with C++20, but rather spend my time with languages where productivity comes first. |
|
I'm happy to see authoring DL models in C# is finally being prioritized. I wanted to share another idea for this just for context. Instead of using TorchSharp one could author models in ONNX format via OnnxSharp (https://nietras.com/2021/03/20/introducing-onnxsharp/) if that had an easy to use API for that. With a proper C# API, not the pythonesque TorchSharp API. Training could then be done via the ONNX Runtime Training API. Or even TorchSharp if that could load ONNX models. If I had time I would be working more on fulfilling this idea. Having both options would be best of course. Important thing is if ML.NET settles on using ONNX format, there are a lot of options. So happy with that too. |
|
Apple silicon support is important |
|
@nietras I am aware of and follow your contributions through OnnxSharp Keep it up!!!
TorshSharp save/load ONNX models has been discussed. We need more people to join in to discuss and support this idea. |
I like this one !! I've asked the same in to retain as option the same native API naming conventions, but they did not listen to me ... ( |
|
Why not to use also https://github.com/SciSharp/TensorFlow.NET ? Also there are Miguel package https://github.com/migueldeicaza/TensorFlowSharp ... but he does not evolve it unfortunately |
|
@redradist |
What ?? When ? Where ?? |
|
@redradist You can get ML.NET book e.g. from Amazon which assist in using Tensorflow.NET in ML.NET |
|
@GeorgeS2019 |
|
@GeorgeS2019 Seems like I've found it ;) |
|
This plan tackles almost every problem that most of us have when it comes to using ML.NET for DL, it is very exciting! |
Why python-like apis then? |
Those are two separate issues. TorchSharp is pure C# code calling into the native engine that powers PyTorch. It also happens to have adopted the shape of APIs of Python, to simplify the job of moving code from Python to C# |
|
Please can Reinforcement Learning be formally added as a goal under the banner of "Deep Learning Support"? I put a ticket on modelbuilder about a year ago, here: dotnet/machinelearning-modelbuilder#1026 |
I support your idea, but most people will feel that it's too radical. And cutting the ecology of torch is a double-edged sword. |
Cutting the C# /. NET ecosystem is even a more dangerous double-edged sword. Writing python and forcing C# developers to understand and use python. They should take a look at what OpenCVSharp is doing. Beautiful apis that conforms to dotnet way of doing things |
|
This is not doing reinforcement learning using ML.NET. The training happens in Python. |
|
@JimFFM I already have a workflow that works well, that makes a round trip through Python [and has a more performant and less byzantine; but also less capable; ecosystem around it]. But I'd really like to skip said round trip and do it directly. |
|
"1. Make it easier to consume ONNX models in ML.NET using the ONNX Runtime (RT)" Could this fit the use cases for model builder?
I think it would make sense to help .NET devs consume ONNX models, even without knowing ML.NET libraries. I believe the data scientist creating the model may not be the one integrating it with the app. Model Builder might help overcome this gap between ML domain and application architecture domain. Maybe same feature could also support creating consumption samples for ML.NET .zip models. Data scientist could build the model in any chosen way and then provide the app dev with the models. |
I get where you're coming from, but I'd urge you to consider the simple reasoning for the design decision. Put simply, given the enormous amount of existing documentation, tutorial and guide content and other collateral already created for PyTorch, keeping the naming and API conventions identical makes all this applicable to the .NET bindings as well. No doubt there'll be an opportunity down the road to make a .NET-first API on top of the work done here, but in the interest of not starting entirely from scratch, this approach makes sense to me. |
|
Honestly I think this might not be a great reason. Assuming there will be many people actually porting python code over to ML.NET, after that is done everyone will have to continue using python naming conventions for no reason. With new people coming to machine learning every day though, I think many people will be confused by the different naming, which makes it look like an interop wrapper around a python library, which it is not. It's its own thing. I'm currently using python for my ML projects, but I've been trying to get into ML.NET hoping for some performance boost but also an easy way to build different frontends quickly, but most of all the static typing. Since we can build dotnet apps with almost no boilerplate and can even write dotnet interactive notebooks now, the IMO few advantages of python rapidly dwindle. I'd immensely prefer the naming also being in dotnet convention, matching the rest of the framework. |
@briacht @luisquintanilla @migueldeicaza These out-of-scope topics defined back in Aug 30, 2021 are now being actively developed.
Sept 2023 Update.NET for pytorch TorchRL using TorchSharp [WIP] |
|
Now TorchSharp's TorchVision and TorchAudio have their own nugets nuget Torchsharp TorchVision nuget Torchsharp TorchAudio |
Sept 2023 Update for Deep Reinforcement Learning |
Plan for Deep Learning in .NET
This past year we've been working on our plan for deep learning (DL) in .NET which is outlined in this issue below.
We came up with this plan by looking at recent deep learning trends, staying in line with current Microsoft strategy around deep learning, and gathering a variety of customer evidence (which you can read about in the Customer evidence section below).
Our goal is to execute this plan by the end of 2022.
Feedback
We would love to hear your feedback about our plan and whether this will fulfill your requirements for deep learning in .NET!
You can leave your feedback directly on this issue.
The plan
The main stages of the plan are:
As part of this plan, ML.NET will continue to be more top-level and scenario focused, while TorchSharp will power the Deep Learning training and features in .NET.
As we iterate and work on the plan, we will make updates to this issue, including linking to relevant issues and PRs.
Stage 1: Make it easier to consume ONNX models in ML.NET with the ONNX RT
In ML.NET, you can already consume pre-trained ONNX models via the ONNX RT.
However, we have seen many issues from customers around the complexity of ONNX model consumption in ML.NET, particularly when trying to figure out the inputs and outputs of the ONNX model (which are required for consumption in ML.NET).
So, we plan to:
Stage 2: Fully support and productionize TorchSharp
TorchSharp is an API that provides .NET bindings for the PyTorch engine.
TorchSharp will be the API-based Neural Network training and inference in .NET; the custom deep learning training that you can do in PyTorch, you will be able to do in .NET with TorchSharp.
As mentioned in the TorchSharp repo, TorchSharp will retain Python naming conventions. This makes it very easy to port PyTorch to .NET TorchSharp as well as to bring Python examples to .NET.
In order to bring TorchSharp to production, we will:
Ongoing support work will include:
Stage 3: Build a bridge between TorchSharp and ML.NET
Once TorchSharp is fully productionized, it will power higher-level ML.NET deep learning APIs which will follow .NET naming conventions and "feel" like .NET.
We will surface the following through ML.NET (powered by TorchSharp):
Here is a diagram which visually outlines the transfer learning part of the plan:
Out of scope
For this plan, it is currently out of scope to port over the PyTorch domain libraries (Torch Vision, Torch Audio, and Torch Text).
Customer evidence
After talking to a variety of .NET customers, we discovered the following:
Additionally, the following issues filed in the repo add further evidence for backing up our deep learning plan:
Image segmentation / regression
NLP
Local object detection
Torch support
ONNX consumption
Other
The text was updated successfully, but these errors were encountered: