.NET Polyglot Notebook and Jupyter Notebook documentation and samples

[dcuccia]

.NET Polyglot Notebooks are a great way of sharing executable "scripts" that require minimal tooling (works cross-platform, right inside Visual Studio Code).

PythonNet is a modern and elegant interop tool that allows calling .NET libraries directly from Python. Compared to our "VTS Wrapper" Matlab interop tools, this tool seems to have a lot less mismatch with .NET objects and arrays/dictionaries, meaning that much less (if anything) would need to be "wrapped" at all - should directly be able to call .NET libraries in a Pythonic style, straight from a Jupyter Notebook. And of course, it would work cross-platform and be free and probably make lots of graduate students happy. :)

I'd be delighted to work on some starter demos of these that would mirror the Matlab demos (and each other). Of course, we wouldn't/shouldn't remove Matlab, just would add. Thoughts?

[lmalenfant]

This is one of our priorities post short-course and we would like to have something in place before the next short-course, if you could start working on this, it would be great.

I think we should put this in a separate repo, what do you think?

Our MCCL and WPF applications are in separate repos and we reference the NuGet package, do you think this will work for this also?

[dcuccia]

Super! Re: separate repo, yes, I think that's a good idea.

Based on using this approach at work, I've found that the only downside is that the Notebooks aren't a great way to develop/debug the scripts. My proposal, based on what has worked for me, is this:

1. Create barebones .cs "scripts" project in the main Vts solution, with a namespace like Vts.Scripting perhaps, and use C# top-level statements combined with a handy "multi-main-function" tool called SmallSharp to mimic what will be in the .NET Polyglot Notebooks. This means that any compilation or runtime issues can utilize the full C# compiler and runtime debugger. The other benefit is that you can optionally run these in an integration test phase within the VTS library CI/CD pipeline (Vts.Test.Scripting), so that we catch any regressions to the scripts that might occur when the VTS library changes, and therefore catch/flag any issues that will necessarily need updates to the libraries in the other repos. Finally, anyone more comfortable working out of the VS or VS Code IDE can use these scripts directly if they desire.
2. Next, mirror these scripts in a separate repo (vts-notebooks?) for .NET Polyglot Notebooks, which will essentially be a copy-paste of what's in the script, but with a) the additional decorations needed to reference the external Nuget package for Vts.dll etc, and b) with some friendly formatted text in-between the code describing what's going on.
3. Finally, mirror these scripts in Jupyter Notebooks. I don't think we'd need a separate repo for this, we could just keep the efforts in parallel folders.

Sound good? Want me to create the tickets/projects and link to this, or would you like to?

[lmalenfant]

Please go ahead and create the tickets/projects and I can create the repo once I'm finished with the release.

[lmalenfant]

Maybe we should do a little work to decouple Matlab from testing MCCL? I'd much rather have a uniform approach to the same type of problem (store in same repo, or in separate repos), than have a patchwork solution. And also generally, decoupling scripts vs core libraries vs apps (MCCL) would help significantly with this project's agility and ability to release without as much ceremony/work.

I will create a separate discussion for possibly decoupling MATLAB from the main repo. Currently it does the MATLAB release and the MCCL release at the same time so decoupling it as is, would make the release more complex not less.

We should continue with the notebook branch separately and not mix these issues together.

[lmalenfant]

@dcuccia My suggestion here is to create another branch with just the code changes and the matlab folder as it was previously. Alternatively if you would like to clean up this branch (revert all the changed MATLAB files and put the folder structure back) we can try and create another branch using a squash commit and make sure the large binary files are no longer in the branch.

[lmalenfant]

@dcuccia I have a created a new issue to separate the Vts, Vts.Scripting and PowerShell changes so we can merge them in before the release and have more time to spend on the notebooks. The Python notebook works great but I am having issues with the CSharp notebook. The paths don't work well and unlike the Python when I get the current directory it is not getting the actual current directory. I think we need more instructions before we release this piece.