F# Support in .NET Native and UWP #1096

Open
NumberByColors opened this Issue Apr 22, 2016 · 40 comments

Projects

None yet
@NumberByColors
Member

We’ve been working on adding F# support to the .NET Native compiler. When this work is finished, you’ll be able to write Portable Class Libraries in F# that can be used from Universal Windows apps, which are compiled with .NET Native. Since the .NET Native compiler is closed source, this issue is here as a way to share status regularly from the Microsoft team.

As some background, .NET Native is a compiler that turns .NET IL code directly into native code ahead-of-time, rather than just-in-time. If you hear “.NET Native doesn’t support F#,” what’s meant is “.NET Native doesn’t support all of the IL produced by the Visual F# compiler.”

Outline of the work required

  1. Identify the specific work required for .NET Native to support F# - In progress
  2. Implement the corresponding bug fixes and new features in the .NET Native and Visual F# compilers - In progress
  3. Implement whatever is necessary in Visual Studio to support source code, projects, and assemblies for F# Portable Class Libraries in Universal Windows apps - Not yet started.

Identifying the specific work required

There are two kinds of issues we’re looking into: issues which we’ve confirmed by observing failing .NET Native compilations of F# IL; and issues we expect to have, but haven’t yet confirmed.

David (@numberbycolors) and Kevin (@KevinRansom) have spent several weeks working to compile the fsharp test suite with .NET Native. Most tests compiled without problems: 18 of the 23 unit tests with the coreclr tag. Several issues have been encountered in the other 5 failing tests:

  • A bug where a 17 layer-deep nested generic type (eg. A<B<C<D<…<T>>>>>) would cause .NET Native to infer the type was infinitely deep and fail
  • A bug where enum members marked ‘internal’ would cause .NET Native to fail.

There are other issues which we expect to encounter, but still need to confirm in other tests:

  • The .tail IL instruction isn’t yet supported by .NET Native
  • Tail call optimization, which is needed for the correctness of some F# programs, isn’t yet supported by .NET Native.

Implement the corresponding features and fixes

The next step is implementing the corresponding fixes and features in the .NET Native and Visual F# compilers. This work will be prioritized against the other work going on in .NET Native and Visual F#. Since the total amount of work is still unknown, we don’t have an estimate for when this F# support will be finished and delivered.

Some of these features and fixes have already been underway for months. While we’ve been investigating F#-specific issues in .NET Native, the team has continued to improve .NET Native. One feature known as “universal shared generics” is likely to have improved .NET Native’s support for F#, even without that being an explicit goal of the feature. Learn more about universal shared generics here.

Implement Visual Studio changes

Once the changes are made to the .NET Native and Visual F# compilers, the Visual F# team will need to do some work in Visual Studio to make sure F# projects behave correctly in Universal Windows app solutions.

@zpodlovics

Please do not forget to do the groundwork for the CoreRT (.NET Core AOT) and the LLILC (LLVM based compiler for .NET Core) project too. The native target application support is important on every platform (eg.: mobile, server).

dotnet/llilc#980

@dsyme dsyme added the discussion label Apr 26, 2016
@lilred
lilred commented Jul 7, 2016

Can we get a status update? :)

@rogersachan

I'd like to hear some of the progress made too

@isaacabraham
Contributor

Note that .NET native has been delayed across the board - it's not even made it for C# yet.

@OnurGumus

That's incorrect. UWP apps must be and are .net native.

@isaacabraham
Contributor

Apologies, my bad. I was sure I read somewhere that it hasn't been bundled with the final release of .net core.

@dsyme
Contributor
dsyme commented Jul 7, 2016

@isaacabraham Both of you are correct. The native compiler for .NET Core is not yet bundled as open source. The native compiler used for UWP apps is proprietary. If I understand correctly they are different native compilers (though many components are shared)

@rogersachan
rogersachan commented Jul 11, 2016 edited

How long has it been delayed for? Will we at least have F# UWP for RS2 in 2017?

@lilred
lilred commented Jul 11, 2016

@rogersachan: I'm under the impression that Microsoft isn't investing too much in F# anymore. Two employees (@NumberByColors and @KevinRansom) make up the bulk of recent contributions to the visualfsharp project, so the F# core team probably isn't much more than two people strong. Adding F# support to UWP is a very large task for two people to tackle.

What I'm saying is that, at this point, any timely resolution of UWP support is going to be coming from the community. It's hard to come up with estimates in those cases.

TL;DR: you want it, you build it. Microsoft doesn't care (for some suitable definition of "care").

@enricosada
Contributor

@lilred the visualfsharp team now is @KevinRansom @otawfik-ms and @cartermp .

@cartermp
Contributor

Update time!

Clearing up .NET Native

.NET Native is, for now, the proprietary native compilation technology used for AoT compilation of UWP apps. No other scenario is supported at this time.

The .NET Native toolchain team is tackling a huge number of core issues that affect everything (not just supporting one particular language), and these are a high priority for them to fix. These issues affect key internal and external customers.

Does the .NET Native Team Care about F#

In a word, YES. The issues which have been found are on their backlog and they have every intention of adding support for them. But to be clear, these are not trivial to support. Tailcall optimization, for example, is a big engineering effort that will take significant time to build and get right.

So where is F# on .NET Native Support Right now?

Still in Investigation Mode. We get recent internal builds of the toolchain that we run tests on, we add tests to try and see what else we can break, and we report issues. These issues get logged into the .NET Native engineering backlog as soon as they come up.

What can the F# community do to help?

The best thing you can do is to ask for F# support (such as UserVoice) and provide examples of how you would like to write F# for UWP applications.

If you have a need to write F# code for UWP apps, please reach out to me! You can email me directly (phcart at microsoft dot com) or DM me on Twitter. I want to get as much concrete info about needs as possible, as I can take these directly to the .NET Native team.

Does Microsoft Care about F#?

Yes! Yes yes yes yes yes! And more love for F# is coming, believe me - and not just in the OSS world. But things are taking time because so many things - the Visual Studio platform, the entirety of .NET - have been making major shifts in direction and huge engineering investments.

Okay, fine, so how about some dates?

As far as UWP/.NET Native support is concerned, I don't have a date for when F# support is completed. It won't be this year, though. As mentioned earlier - for example, getting tailcall optimization fully supported is going to be nontrivial and will take some time.

I'll reiterate this - F# is on the .NET Native roadmap, smaller issues have already been worked on, the big issues are logged, and the .NET Native team intends to fully support F#. But .NET Native and UWP are far from done, and there are massive engineering investments yet to come.

Hopefully this clears a few things up. Thanks for being understanding. If F# support for building UWP apps is essential to you, please send me at email (or DM me on Twitter) so we can set up a chat.

@OnurGumus
OnurGumus commented Jul 11, 2016 edited

Thank you for very detailed answer. Much needed and appreciated. However I have few opinions and questions.

  1. Some part of the tail call optimization is done directly by the compiler itself rather than JIT. How about adding .net native support to F# and implementing tail call support later ? Worst case we will eat the stack excessively, the loops will be slower or get a stackoverflow, I can live with this till the implementation complete?
  2. How about .NET native optional for F# projects? The only major drawback is it won't run on HoloLens and XBOX but considering there are very few UWP targeting these platforms we can live with it.
  3. How about asking assistance from xamarin guys ? Since they have already solved this problem. Remember IOS also disallow runtime code generation and xamarin does support F# iOS apps via AOT.
@sefffal
sefffal commented Jul 11, 2016

.NET native is only a requirement of the windows store no? In the meantime, could we start creating UWP apps with F# that can be sideloaded or distributed through some other means?

@dsyme
Contributor
dsyme commented Jul 11, 2016 edited

@lilred It's helpful to distinguish between the Visual F# IDE tools, the Xamarin tools and the F# language itself. Along with many from the community, people at Microsoft contribute to the language (I've recently submitted about 20 PRs to this repo, and I work for Microsoft), some contribute to .NET Core support, some to the Visual F# IDE tools, some to the Xamarin tools, some to Azure Functions support for F#, some to data science tools for F# (e.g. Tomas Petricek is consulting with Microsoft and contributing to FSharp.Data and FsLab) etc. There is also have a very steady stream of contributions to the Visual F# docs, including many from community.

That said, I think in the modern word of .NET open source, the most crucial roles for Microsoft are to enable high-quality open source contributions, amplify the base of contributors and package/deliver the software to enterprises. It's very noticeable that the areas that are closed-source such as UWP Native can become (understandable) sources of frustration, because it's plausible that the F# community could "just go fix it" if these components were open.

@ReverseBlade Yes, it's feasible for tailcalls just to be ignored in first-cut support. UWP Native would just have to ignore the tailcall prefix.

@OnurGumus
OnurGumus commented Jul 11, 2016 edited

@sefffal it is only required for publishing to windows store. For side loaded UWP apps, you can freely use F# and it will work on desktop, tablet and phone (but not on platforms that require .net native like hololens or xbox). (just make sure you uncheck .net native in build options)

@rogersachan

So we won't see anything until VS "15" I suppose (seeing as VS vNext won't be ready anytime soon), but it's good hearing that it'll be done right and not rushed through like .NET Core's implementation of F#.

@lilred
lilred commented Jul 11, 2016 edited

I know this is on a tangent, but does anyone have information as to why the UWP compiler is being kept closed-source? It seems like such a hurdle to developers. It's not like Microsoft are actually selling the compiler, they give it away as part of the Windows 10 SDK.

@dsyme
Contributor
dsyme commented Jul 11, 2016 edited

not rushed through like .NET Core's implementation of F#.

This support is still in alpha ( latest docs here ), so I don't think it's being rushed (if the current bits were labelled as RTM quality I'd agree). However what is happening is that it's being done in the open.

I understand that some traditional users of Microsoft's dev tooling don't appreciate it, but seeing the cake made (and contributing to it) will be more normal now.

@exercitusvir

@cartermp Thanks for the update.

What can the F# community do to help?

The best thing you can do is to ask for F# support (such as UserVoice) and provide examples of how you would like to write F# for UWP applications.

If you have a need to write F# code for UWP apps, please reach out to me! You can email me directly (phcart at microsoft dot com) or DM me on Twitter. I want to get as much concrete info about needs as possible, as I can take these directly to the .NET Native team.

I am not sure what you are asking for exactly. There are over 1500 votes for Add F# support for .NET Native and there are over 2000 votes for F# support in .Net native for UWP with examples why people would like to develop UWP apps in F#. There are more votes for .NET Native support for F# than any other F# language feature request on uservoice. This is probably almost the entire F# community that signed up for user voice. Why do you ask for concrete examples? F# is a .NET language that produces valid CIL and .NET Native needs to support all valid CIL to be called ".NET". Otherwise it should have been called C# Native or VB Native.

If Microsoft was really behind F# as much as you claim, then it would simply put enough resources behind it to make it work. Same with .NET Core. Talk is cheap. Show us some action.

@dsyme
Contributor
dsyme commented Jul 11, 2016 edited

@exercitusvir For .NET Core, the most positive way to help is to contribute to the support. There is a list of known issues, though perhaps more important is trialling the existing support and reporting problems. Another significant issue is cross-compiling on Linux and Mac.

@enricosada
Contributor

Another way to help is to contribute current work in progress so vf# team can concentrate on the part we cannot do ( closed source, coord with other teams )

For example there is some to do to finish the dotnet core, lot of testing and cleanup

There is also corert and illc, that's the aot compiler, and it's oss, as alternative to the closed source .net native. coreclr too has open issues.
Other communities implement that themselves, if it's oss.

Obv it's also possible wait and ask for more developer hours, but ihmo I think this is not the right place, because here everyone agree about that. The commercial side can help about that, but nobody of them is reading these issues

@Mike-EEE

👍 to @exercitusvir. Truly amazing to see how long it has taken to get F# "support," and that UWP was designed in such a way that it didn't even consider it from the outset. Truly terrible and costly oversight, and reflects poorly on the entire UWP organization.

Reading over the language used here, I have to say that it really doesn't sound like you are committed to F# or really have very definite plans to support it. It really sounds like a go-around, to be honest. I hope I am wrong, but I for one am not getting the warm-and-fuzzies here, and I know there are others in the votes above that are expressing that, too. The fact that you are still caught in a quagmire of closed-source considerations and old-skool roadblocks really says a lot and is not something you see from other MSFT groups these days. We're also approaching the 6-month mark of this issue being created, and it doesn't seem like there is at least an alpha/test pilot, yet? Or really, any dates to watch/expect output. Concerning.

I also second the fact that you want to pull in developers for private conversations rather than listening to the thousands who have already spoken. This not only seems like you are trying to delay/fidget your way out of this, but makes me question which data you are leaning on to make your decisions on in the first place. Because clearly UWP management got this wrong. UWP group as a whole has a very bad rep of not being very community-centric, engaging, and transparent like other groups are, and it has done nothing but been a disservice for everyone involved.

But truly, the greatest disservice we are all saddled with is that somehow, someway, a Microsoft-managed group "managed" to build a ".NET" technology without the ability to support all .NET languages like the original architects of .NET designed and intended. This whole debacle should be a non-issue, yet, here we are. Good to see all those infamous, iron-clad whiteboard riddles you dole out to perspective interview candidates designed to really only filter the best-of-the-best are working their wonders. 😛

Hope this isn't too harsh, as ultimately I do support your efforts -- you've got my upvote and reactions! Just speaking my mind here -- and yes, that does get me into trouble, LOL!

@OnurGumus
OnurGumus commented Sep 15, 2016 edited

@Mike-EEE , actually the solution is very simple. Either, They will ignore tail calls. And fix Generic support,
or make .NET native optional for Phones and PCs at least for F# projects.
But all is not lost. The number of F#ers are growing and you can develop Xamarin apps with F# that runs on ios and android. Since no one actually uses UWP at all, who cares.

@Mike-EEE

Since no one actually uses UWP at all, who cares.

Yeah... about that. 😛 Really starting to think UWP is best relegated to simply a rendering platform to render OTHER application UI built by teams who actually understand and care about .NET (like Avalonia).

@forki
Contributor
forki commented Sep 15, 2016

Meanwhile in community land: fable-compiler/fable-react_native-demo#10

@OnurGumus
OnurGumus commented Sep 15, 2016 edited

Actually we started to use Fable instead of TypeScript. I am so delighted with it. It's the best thing happened on javascript for the NET community. And I am sure vast majority of Microsofters never heard of it.

@jkotas jkotas referenced this issue in dotnet/corert Oct 25, 2016
Open

F# support #2057

@charlesroddie

Any progress @cartermp ? My startup is using F# / Xamarin Forms and we want to target UWP.

@cartermp
Contributor

@charlesroddie Unfortunately, none which could be directly actionable for you at this time. F# Support for CoreRT, the cross-platform native-compile runtime is being tracked and worked on (which is the first step towards support), but the ability to use F# in a UWP app likely won't be available for some time.

@Mike-EEE
Mike-EEE commented Dec 12, 2016 edited

but the ability to use F# in a UWP app likely won't be available for some time

Wow, I'm trying to be supportive here, but this has already been in the works for how long now? Maybe it's time to make UWP open source like the rest of MSFT? Seems like it would get done a lot faster. 😛

@OnurGumus
OnurGumus commented Dec 12, 2016 edited

@charlesroddie Let me inform you a bit more. First of all, you can surely target UWP with F# libraries by using portable class libraries. The fact is it will only not work when you turn on .net native. Simply don't use .net native and you are fine.
The problem is Windows Store unreasonably enforces .NET Native. So you won't be able to publish your app to Windows Store, but it will work on a tablet, pc and phone. But you will be able to publish it to Apple Store and Google Play ironically.
It won't work on XBOX nor HoloLens I guess this is not a problem.

The real issue here when Microsoft invented .NET Native, they checked out F# usage and found out that only 0.01% of Windows Store apps make use of F#. So they ignored it. The solution is simple though:
1-) They could have made .NET Native optional, (it won't work on xbox one or HoloLens but will work else where)
2-) They could have ignored tail calls and in depth generics as a first step and fix them later.

For some reason they choose not to. Plus the resources given for F# tooling is rather limited from Microsoft side. That is actually contradictory , considering they are adding some functional features to C #7 , though those features are rather lame IMHO. So to sum up , Microsoft doesn't get functional programming is the future whereas entire world started to use functional paradigm actively even in the front end, now there are frameworks like Elm, Apple moved to Swift, Scala in Java world is a true first class functional language. Most likely Microsoft will follow in couple of years later with some regret.

Enough ranting, here's your options for F# usage.

  • Use F# with UWP but don't publish it to windows store but only iOS and Android.
  • Use F# with Xamarin and ignore UWP considering how UWP is not widely adopted.
  • Use F# with Fable framework to compile it to JavaScript (it actually works very fine even for complex F# code) then use it via from React or UWP JS.
  • You can also use unity 3d for gaming with F# except for Microsoft platforms beside desktop.
@Mike-EEE

Use F# with Xamarin and ignore UWP considering how UWP is not widely adopted

Wise words that get wiser by the day, @OnurGumus.

@martinsuchan

when Microsoft invented .NET Native, they checked F# usage and found out that only 0.01% of Windows store apps make use of F#

Maybe the reason for the low usage is the fact, that there was very poor support for F# code in Windows 8/8.1 Store apps, only limited subset of API was available in PCLs.
I really wanted to use F# for my Windows Store apps, use it for Page/Controls code-behind, etc. but it was just not working last time I tried it, and it looks like it's becoming chicken/egg problem now.

@Mike-EEE

Maybe the reason for the low usage is the fact...

I am personally more concerned that a MSFT group would make efforts to limit the number of supported .NET languages used within a .NET technology (regardless of how used/underused that languages is). That seems to run counter to one of the fundamental qualities and tenets that has made .NET so great.

@charlesroddie

@OnurGumus I agree very much. Either 1. (optional .net native) or 2. (allow a basic .net native for F# and improve over time) would be very good for us. If the current approach (sort out .net native fully and then allow f#) continues to take a long time, I hope MS will consider allowing 1. or 2. until full .net native support is ready.

@cartermp So CoreRT is linked to .NET native, so getting CoreRT to work will help get .NET native to work?

@Krzysztof-Cieslak
Member

Or just use Fable + React Native /shrug

@forki
Contributor
forki commented Dec 12, 2016

+1000 Fable |> React Native is really really great.

@cartermp
Contributor
cartermp commented Dec 12, 2016 edited

@charlesroddie While it's not true that CoreRT is a part of .NET Native, getting support on CoreRT means getting support for native compilation such that it could be worked into the .NET Native toolchain. There are a lot of shared concepts and problems to be solved (e.g., support for tailcalls, how to handle reflection, F# enums, etc.).

From a larger standpoint, CoreRT support also means getting support for cross-platform, native-compiled F# applications.

@OnurGumus
OnurGumus commented Jan 15, 2017 edited

Please sign and spread the word to fellow F# devs:
Change.org: Make Visual F# a true first class .NET language with proper tooling and UWP support
https://www.change.org/p/microsoft-make-visual-f-a-true-first-class-net-language-with-proper-tooling-and-uwp-support?recruiter=664638536&utm_source=share_for_starters&utm_medium=copyLink

@rogersachan
rogersachan commented Jan 15, 2017 edited
@psfblair

As a HoloLens developer, I'm pretty miffed to read people's dismissive attitudes to the effect that it's no big deal if HoloLens isn't supported. It is a big deal. This platform is going to be a huge draw of new developers to the Microsoft community. People who want to be on the cutting edge want to use cutting-edge tools.

For those who want to develop 3D apps for HoloLens there is a workaround. Unity 3D comes with two "scripting backends" for incorporating developers' .NET code into a Unity3D Windows Store app for HoloLens. One of these scripting backends is called "IL2CPP." When it is used, the IL in .NET assemblies used in the Unity project -- including F# assemblies -- is translated into C++, which is then compiled to .NET native code.

(The way to get F# code into a Unity project is to compile the F# code into a DLL, and have a post-build step copy the DLL and the FSharp.Core DLL into the Assets\Plugins folder in the Unity project.)

I tested a tail call to see what would happen. So this code:

  let rec callTailRecursiveFunction index =
      if index <= 0 then
          ()
      else
          index - 1 |> callTailRecursiveFunction

turns into this:

extern "C"  void MyClass_callTailRecursiveFunction_m1362434375 (MyClass_t1938649317 * __this, int32_t ___index0, const MethodInfo* method)
  {

  IL_0000:
      {
          int32_t L_0 = ___index0;
          if ((((int32_t)L_0) > ((int32_t)0)))
          {
              goto IL_0006;
          }
      }
      {
          return;
      }

  IL_0006:
      {
          int32_t L_1 = ___index0;
          ___index0 = ((int32_t)((int32_t)L_1-(int32_t)1));
          __this = __this;
          goto IL_0000;
      }
  }

Which seems to indicate that, in this case at least, tail calls are optimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment