Add support for referencing .NET projects in F# interactive

[toburger]

Is your feature request related to a problem? Please describe.

It would be nice to have similar functionality like in other languages, where it is possible to reference a project inside an interactive session.
Languages that support such functionality are for example: Haskell, Elixir, Python, Clojure

Describe the solution you'd like

I would like to start an interactive session and reference the project like I would reference an assembly file today.

An example could be (F# projects only):

 #r "fsproj: MyProject.fsproj"

or (across languages):

 #r "project: MyProject.fsproj"
 #r "project: MyProject.csproj

Those two proposals need to be tackled very differently. For a F# only solution it is enough to reference all library references and the F# source files. In order to work across .NET languages it is needed to compile the project first and then reference the libraries and the generated assembly.

@Krzysztof-Cieslak has already created a prototype where he got working a F# only solution: https://github.com/Krzysztof-Cieslak/DependencyManager.FsProj

So the first question to answer is:

• Should the solution work for fsproj and csproj files from the beginning?
• Should we define a cross project syntax and start with an F# only solution?
• Is it a F# fsproj only thing?

Describe alternatives you've considered

• Reference every library reference and every project source file by hand.
• Use #r: "nuget: Dependency" for every dependent library and add every source file by hand
• Use paket (generate_load_scripts) to generate library references in a script file that can be used in your script and then reference every source file by hand

Additional context

I've created this issue because there was an ongoing discussion started here: fsharp/fslang-suggestions#542

[abelbraaksma]

Considering that projects can be large and take a long time to compile, I would suggest we make this work with the compiled output from the referenced project. That way, it's likely to be much easier to implement (basically, the path should be added to the assembly search paths, and the output exe or dll should be loaded).

If the dirty flag is set, perhaps a warning could be given.

BTW, I remember there used to be a Powertools VS extension that did something like this, but I think it doesn't exist anymore.

[toburger]

BTW, I remember there used to be a Powertools VS extension that did something like this, but I think it doesn't exist anymore.

Right! Last supported Version of Visual Studio was 2015: https://fsprojects.github.io/VisualFSharpPowerTools/generatereferences.html

[toburger]

I've written an alternative prototype which first restores and compiles the project and references the generated assembly instead of the source files: https://github.com/toburger/DependencyManager.FsProj

With this approach I was able to load a more complicated project of mine into F# interactive.

[cartermp]

@toburger nice work! This would be a great thing to add for .NET 5. It could end up getting pretty tricky, though, since we'll want to support more than just F# projects. @KevinRansom @jonsequitur what do you think?

[jonsequitur]

I think this is great. I'd like to throw out some related ideas and get people's thoughts.

Our discussion around #r "project: (spanning both F# and C# via .NET Interactive) has included a few different potential goals:

• Use the project's code and references from the interactive session.
• Consistent behavior between C# and F#.
• Cross-language support, e.g. using `#r "project:/path/to/some.fsproj" from within C# interactive code
• Interactive debugging of the target project.
• Loading dotnet-interactive kernel extensions from within the target project or from its dependencies.

Some of these will require using the interactive dependency manager that @KevinRansom has been building, as it provides more information about packages than you can get from a vanilla build.

[Happypig375]

What about vbprojs?

[toburger]

• Use the project's code and references from the interactive session.
• Consistent behavior between C# and F#.
• Cross-language support, e.g. using `#r "project:/path/to/some.fsproj" from within C# interactive code

The only F# project specific API in the prototype is to load the project information (files, references, target path).
The compilation itself is done via FAKE which in turn calls dotnet restore/msbuild which should work with any .NET project (even VB 😉).

• Interactive debugging of the target project.

For me personally debugging has a very low priority in scripting. 😄

• Loading dotnet-interactive kernel extensions from within the target project or from its dependencies.

Can you explain more on this?

Should we also limit the target projects to SDK-style projects only?

[zyzhu]

Interactive debugging in data science scripting is important when project gets complex.

[toburger]

Interactive debugging in data science scripting is important when project gets complex.

I would argue that debugging is more important in scenarios where mutation is the dominant paradigm. But given that you are the maintainer of Deedle I take my statement about debugging back. 😉

[cartermp]

There's some additional considerations here, since the notion of "referencing a project" is very complicated.

• Currently the extensibility mechanism is .NET Standard 2.0, so it works on desktop .NET Framework FSI. Does that also apply to this feature?
• Are non-.NET SDK-style projects loaded? If so, which kinds? (yes, these are all distinct from each other)
  • Legacy F# projects
  • Legacy C# projects
  • Legacy VB projects
  • Legacy C++/CLI projects (Windows only of course)
  • Flavored C#/VB projects (e.g., Xamarin)
  • Legacy ASP.NET MVC projects
  • Legacy ASP.NET Web Forms projects
• Are C++/CLI (on .NET Core) projects supported?
• Are .NET Framework projects that use the .NET SDK supported?
  • If running on .NET Core, what happens when a .NET Framework project is referenced? At least on Windows, .NET Core can load some .NET Framework projects for compatibility purposes, so this could work in theory
  • If running on desktop FSI, what happens when a .NET Core project is referenced?

[Happypig375]

@cartermp Isn't that distinction part of what MSBuild does so we don't have to do the work ourselves?

[cartermp]

Depends on which MSBuild is involved 🙂

[cartermp]

There's also more to consider than just MSBuild here. We're still dealing with subtleties around native assemblies and RID-specific configurations with the dependency manager today, for example.

[ntwilson]

In my solutions with multiple F# projects, I usually have a Packages.fsx script in the root folder that pretty much duplicates paket.dependencies and references them all either via #r nuget or just referencing the DLLs directly from a bin/ folder, and then another script for each project that has #r for the project DLL + the DLLs of all referenced projects that aren't on nuget (working in a solution of multiple projects). Is this the recommended approach if you want to load your project into FSI or is there some simpler method? It can end up being a bit of work to manage all of the scripts and I often find them out of date.

[onionhammer]

I don't think there is a recommended approach - just a massive gap in tooling support.

[jwosty]

@ntwilson that doesn't sound like a terrible approach, and could probably be automated with a small script that traverses the project file and generates a Project.fsx

[inouiw]

I thought adding a fsx file to generate testdata would be nicer then to write a unit test but just referencing the dll of the main project seems not enough. The Compiler is asking for missing references of types defined in nuget references.
The proposed solution with #r "project: MyProject.fsproj" would be great.

[inouiw]

For everybody who is looking for the references that must be included in a fsx file to call code in a .net project. In VsCode in the "F#: Solution Explorer" of the "Ionide for F#" extension you can right-click on a project and select "Generate references for FSI". Then a referencs.fsx file is generated with many #r and #load lines. 625 lines in my case.

[ThisFunctionalTom]

A few months ago I used the PoC project form Chris and Ionide.ProjInfo and implemented the functionality. The implementation is here: DependencyManager.FsProj, and in the nuget README is explained how to get it working.

It is deployed as a dotnet tool to have all the dependencies in same directory so when registering new dependency manager in fsi everything works.

It could also be used as a dotnet tool to generate load script for fsi if the setup is too complicated.

I didn't test it extensively but it worked for a few samples I tried. I didn't really used it that much in the end 😅

I am not sure how to move it to fsharp because, I guess, adding Ionide.ProjecInfo as dependency to fsharp is not a good idea. Maybe we could extract the needed functionality?

[fradav]

A few months ago I used the PoC project form Chris and Ionide.ProjInfo and implemented the functionality. The implementation is here: DependencyManager.FsProj, and in the nuget README is explained how to get it working.

It is deployed as a dotnet tool to have all the dependencies in same directory so when registering new dependency manager in fsi everything works.

It could also be used as a dotnet tool to generate load script for fsi if the setup is too complicated.

I didn't test it extensively but it worked for a few samples I tried. I didn't really used it that much in the end 😅

I am not sure how to move it to fsharp because, I guess, adding Ionide.ProjecInfo as dependency to fsharp is not a good idea. Maybe we could extract the needed functionality?

@ThisFunctionalTom would you like to update it for net7.0 (currently broken with new fsac and net7.0)? I tried to make this update myself, but the projects are not loaded anymore, logging the loader says it is missing obscure dll with ChangeWaves.

The ability to load a project into .fsx is essential for my workflow (experiment interactively with a WIP library and backport experiments in the library) and the current way of doing it (relying in the "generate references for the project" is a little cumbersome and error prone, so doing it at fsac/fsi level is a real boon. (but doesn’t work anymore, as I said, unless I come back to net6.0 and to old ionide versions).

[ThisFunctionalTom]

I am on holidays this week. I hope I can find some time next week to look into it.

…

On Wed, 14 Jun 2023, 19:57 François-David Collin, ***@***.***> wrote: A few months ago I used the PoC project form Chris and Ionide.ProjInfo and implemented the functionality. The implementation is here: DependencyManager.FsProj <https://github.com/ThisFunctionalTom/DependencyManager.FsProj>, and in the nuget README <https://www.nuget.org/packages/DependencyManager.FsProj#readme-body-tab> is explained how to get it working. It is deployed as a dotnet tool to have all the dependencies in same directory so when registering new dependency manager in fsi everything works. It could also be used as a dotnet tool to generate load script for fsi if the setup is too complicated. I didn't test it extensively but it worked for a few samples I tried. I didn't really used it that much in the end 😅 I am not sure how to move it to fsharp because, I guess, adding Ionide.ProjecInfo as dependency to fsharp is not a good idea. Maybe we could extract the needed functionality? @ThisFunctionalTom <https://github.com/ThisFunctionalTom> would you update it for net7.0 (currently broken with new fsac and net7.0. I tried to make this update myself, but the projects are not loaded anymore, logging the loader says it is missing obscure dll with ChangeWaves) The ability to load a project into .fsx is essential for my workflow (experiment interactively with a WIP library and backport experiments in the library) and the current way of doing it (relying in the "generate references for the project" is a little cumbersome and error prone. — Reply to this email directly, view it on GitHub <#8764 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZJO4TO72TQC45MTNUGEBTXLH3P7ANCNFSM4LQ26BIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ThisFunctionalTom]

So, I had an idea how to remove all the dependencies of the framework version (by using dotnet list package --include-transitive).
The load script generation part works but something strange is happening when I alt+enter it in ionide. On some simple projects it works as expected but on mostly all projects something crashes and the ResolveDependencies method is called again and again and I have to terminate the fsi.
I already asked for help, hopefully we can resolve it soon.

[nojaf]

Hi Tom,

I took a peek at your fork earlier today. And I have some open questions (also for @vzarytovskii):

• Should it be fsproj: or more general proj: (or project:) and support csproj or vbproj as well?
  I'm just asking if there is a PR, what flavour would the team accept?
• How should this work next to any existing #r "nuget:, I can see both dependency managers fight to resolve FSharp.Core for example. Can these be combined? If so, in what order?
• Should a design-time build happen for the referenced project?
• Do we care about any older types of projects? Or do we stick to the modern SDK ones?
• Does anything change when we encounter Microsoft.NET.Sdk.Web for example? I can imagine we might not want to try loading Microsoft.NET.Sdk.WindowsDesktop in Linux for example.

My main question really boils down to: what would the team accept as PR?

[vzarytovskii]

Hi Tom,

I took a peek at your fork earlier today. And I have some open questions (also for @vzarytovskii):

• Should it be fsproj: or more general proj: (or project:) and support csproj or vbproj as well?
  I'm just asking if there is a PR, what flavour would the team accept?
• How should this work next to any existing #r "nuget:, I can see both dependency managers fight to resolve FSharp.Core for example. Can these be combined? If so, in what order?
• Should a design-time build happen for the referenced project?
• Do we care about any older types of projects? Or do we stick to the modern SDK ones?
• Does anything change when we encounter Microsoft.NET.Sdk.Web for example? I can imagine we might not want to try loading Microsoft.NET.Sdk.WindowsDesktop in Linux for example.

My main question really boils down to: what would the team accept as PR?

I would delegate it to @KevinRansom, since he has bigger picture for dependecy manager and interactive in mind.

[smoothdeveloper]

How should this work next to any existing #r "nuget:, I can see both dependency managers fight to resolve FSharp.Core for example. Can these be combined? If so, in what order?

I think in context of FSI, it doesn't matter, FSharp.Core is always going to be loaded already, here is a reference that could be useful:

https://github.com/fsprojects/Paket/blob/513ca8eebda4451ae47e56ca473e5bc035c5be6d/src/Paket.Core/Installation/ScriptGeneration.fs#L52-L60

there is a hedge case about --noframework to consider though.

Should it be fsproj: or more general proj: (or project:) and support csproj or vbproj as well?
I'm just asking if there is a PR, what flavour would the team accept?

If it is going to be "shipping by default" and work for dotnet interactive / polyglot notebooks (for other IL languages that it supports as well: C# & F# for now), I think it is worth to extend the effort, assuming all the plumbing, etc. relies on msbuild, not something that needs to be maintained.

I'd suggest msbuild: or project:, finding myself the first one more explicit about what is going.

I don't think we should turn "proj" into a noun, and given the low character count of invocation versus what it achieves, I don't feel shortenning the extensions is the thing to optimise for.

Do we care about any older types of projects? Or do we stick to the modern SDK ones?

hoping this is handled through msbuild infrastructure so we don't take responsibility on it in the implementation details managed here.

If this introduces a challenge, I'd drop it, people can resort to build the project and load the binaries.

Does anything change when we encounter Microsoft.NET.Sdk.Web for example? I can imagine we might not want to try loading Microsoft.NET.Sdk.WindowsDesktop in Linux for example.

I also could see issues with some projects that require msbuild rather than dotnet build, so when we start facing edge cases, we can consider how to extend the syntax for the extension, so after the project file name, we could pass extra information.

[KevinRansom]

@nojaf ,
we have debated this feature on and off for ever. It is unlikely to ever get to the quality where we would add it into the product.

#r "nuget: " and #r "paket: " are giant hacks. They are unbelievably useful giant hacks, doing something that is absolutely necessary: I.e. there is no other real way of introducing dependencies to a script that is being typed by the developer in real time, so even with its flaws we continue with it, However, the implementation of shelling out to msbuild executing a restore process and then gathering the dependencies is so lame that the C# team, said ... no way is that ever going anywhere near our beautiful language. Fortunately, in notebooks we were in a position to use the package manager as a library invoked by a pre-processor which enabled dotnet interactive to add the package manager to C# notebooks, although the C# guys always gave me the side eye and muttered about needing to a proper implementation :-).

The scripting interface is really ... really limited it is basically:
#r "somestring" formatted as 'package manager id' 'colon' followed by the rest of string which is owned entirely by the Package manager"

All that command executes the script, and loads assemblies from a list of assemblies and scripts from a list of scripts. Once the package manager has executed the command it will see the string every subsequent invocation.

Because of these limitations we still don't have a mechanism for providing credentials for private nuget feeds, a scenario we get asked about a lot ... and tbh absolutely need to support.

The difficulties of a packagemanager fall into a bunch of buckets:

• It has to work in the ide, which knows nothing about it:
  • So intellisense will be a struggle
  • It can run a lot and easily harm IDE performance - which we prefer not to happen
• It has a very limited integration into FSI string in a list of dlls to load and a list of scripts to load out.
• There are very few places it can exist on disk and be found --- I.e. we need and owe a decent extension point

If it were up to me, I would design a Third party library for managing dotnet projects from scripts, with an object model for accessing build properties and item collections building individual targets and manipulating projects, almost certainly supporting solutions. If you did that you would less restricted by the FSI integration point and have more freedom to implement the many ... many enhancemets, functionalities you would want to add as soon as you had the basic shape in place. I imagine that Fake has already got some basic dotnet project file support, although it has been close to a decade since I really looked at it.

If you still want to go ahead with the #r "projfile: ... " #r "nuget: is extensible, paket exists after all" we should probably do some work to make the nuget global and local tools extension mechanism work, they didn't exist yet when #r "nuget: and #r "paket : were originally created. Indeed we should do something about that anyway to make #r "paket: installation more straightforward. I have been promising that to @cloudRoutine for more years than I am comfortable with.

In short:

• A library would be much more flexible and scriptable than #r "projfile:
• Installation difficulties need to be overcome.
• We would be unlikely to add it to the shipped product, project files are just too varied and unpredictable in their behaviors.
  E.g.
  #r "projfile:VisualSharp.sln /p:Debug" would take minutes to run, and how are you going to display the build spew for example.
  and which of the many build targets would you load, and why? what is the point of loading the net7.0\fsi.dll

[onionhammer]

I don't get what the csharp folk think is the usecase for CSI if they're not going to fully complete this loop. It's a half-baked thing today, and if they want to encourage data science / AI / etc to use notebooks, they should also enable project support. Notebooks are toys not tools without being able to load projects and their dependencies.

It would be insanely useful to be able to use your project's model & data repository layers to build interactive notebooks to explore your data with your existing code, but loading those is just so tedious now without being able to load projects & their dependencies. Notebooks feel like a giant hack in general right now.

[ThisFunctionalTom]

Hi Tom,

I took a peek at your fork earlier today. And I have some open questions (also for @vzarytovskii):

Reading other comments here I just realized that I underestimated the complexity of this very, very much 😅 .
I have a feeling now that the amplifying f# session will probably be more of a discussion what can and if it should be done.

Below I my answers how I thought it could be done, but I am not sure anymore.

• Should it be fsproj: or more general proj: (or project:) and support csproj or vbproj as well?
  The solution now uses dotnet list package and dotnet list reference to get the list of referenced projects and list of resolved nuget packages (the projects must be restored). With this Information I build a fsx file with #r "nuget: references which I hope will be then sent to nuget dependency manager. All projects are currently parsed with simple .NET xml parser and searched for <Compile.. xml tags (I know this is naive implementation but I thought it is enough to start with).

All source files and the generated script file with nuget references is then returned from dependency manager.

So to actually answer your question, with this solution only fsproj files can actually be referenced because only .fs files can be loaded by the fsi.

I'm just asking if there is a PR, what flavour would the team accept?
I think we should discuss this if the solution above is the right way to go. Maybe it is the wrong way?

• How should this work next to any existing #r "nuget:, I can see both dependency managers fight to resolve FSharp.Core for example. Can these be combined? If so, in what order?
  As the fsproj dependency manager generates script with #r "nuget: lines this should actually be handled from the nuget dependency manager. If a project with incompatible nuget references is given, I guess, it will probably fail.

• Should a design-time build happen for the referenced project?
  As the sources files are directly returned from the fsproj dependency manager, no build is needed here, just a dotnet restore so that the right nuget package versions are return from dotnet list package

• Do we care about any older types of projects? Or do we stick to the modern SDK ones?
  The current solution works only with new projects.

• Does anything change when we encounter Microsoft.NET.Sdk.Web for example? I can imagine we might not want to try loading Microsoft.NET.Sdk.WindowsDesktop in Linux for example.
  I must admit, I haven't thought about it. I just wanted to make a simple solution which works in some special cases like developing a library and trying it out live in fsi. I thought once there is a simple solution we can improve it to cover more and more different scenarios.

My main question really boils down to: what would the team accept as PR?
That is exactly what I would like to discuss in amplifying F# session because I want to learn what is necessary for a good PR.

[nojaf]

Hi @KevinRansom, I appreciate your valuable input!

Considering the potential impact on user expectations, it wouldn't be fair to overload the team with this task. Even though the original PR can come from the community, the team would be burdened with maintaining it.

In the short term, I believe it would be worthwhile to consider a community project as an alternative solution.

@ThisFunctionalTom, let's discuss this further next week and evaluate its feasibility.

[jkone27]

this is still amazing if achievable, also and especially for .csproj interop! news on this topic, will it make it to F# 9? 💜

[DamianReeves]

@ThisFunctionalTom, let's discuss this further next week and evaluate its feasibility.

@nojaf / @ThisFunctionalTom did anything come of this? We have .NET 8 now and the community supported approach is not working.

[ThisFunctionalTom]

Hi, I will look into it on the weekend. Sorry if it is not maintained faster, but as the solution is not perfect I am not using it myself and so I always forget to update it to the newest .NET SDK.

[DamianReeves]

Hi, I will look into it on the weekend. Sorry if it is not maintained faster, but as the solution is not perfect I am not using it myself and so I always forget to update it to the newest .NET SDK.

Thanks for that. And no apologies needed, as is the nature with community projects, this is all a labor of love done in your free time. I appreciate the great tool you have provided.

[ThisFunctionalTom]

@ThisFunctionalTom, let's discuss this further next week and evaluate its feasibility.

@nojaf / @ThisFunctionalTom did anything come of this? We have .NET 8 now and the community supported approach is not working.

I deployed a new version 0.2.9. I hope this works for you.