Skip to content
This repository has been archived by the owner. It is now read-only.

F# Status on CoreRT and UWP #6055

Open
charlesroddie opened this issue Jul 5, 2018 · 27 comments
Open

F# Status on CoreRT and UWP #6055

charlesroddie opened this issue Jul 5, 2018 · 27 comments

Comments

@charlesroddie
Copy link
Contributor

@charlesroddie charlesroddie commented Jul 5, 2018

[2018-12-06 update] F# UWP apps can be released on the Windows Store.
#6055 (comment)


Recent testing has assessed the compatibility of F# with CoreRT and .NET Native. Only minor issues have been found, which are easily worked around (see specific issues below).

CoreRT issues can always be tracked and worked on here, and .NET Native issues can now also be tracked now that users can compile using it (See gatekeeper.xml section).

This post is a documentation of the status of F# and CoreRT as requested by @dsyme. This post will be kept up to date. One purpose is to inform F# developers using CoreRT and .NET Native. The second purpose is to inform the request for allowing F# usage on the Windows Store, a 1 line change that may require approval.

Testing

CoreRT and .Net Native went through basic tests (the F# cheatsheet), and specific tests (tail calls and highly nested generics as recommended by @MichalStrehovsky, and some of the F# test suite). .Net Native was further tested in a production app using typical F# structures (a symbolic algebra engine using large discriminated union trees, standard usage of record types, reactive UI using Xamarin.Forms, SQLite). CoreRT is used by a number of F# users already. (@ChrisPritchard has written a number of games without encountering any issues.)

Specific Issues

Issue UWP (.Net Native) CoreRT Workaround
sprintf Fails Can work for some types using rd.xml Use concrete string methods like String.Format.
.ToString() on discriminated unions and record types Fails Fails Override .ToString() on these types.
LINQ expressions Fails Fails Use an alternative to LINQ (e.g. SQL) or use LINQ from C#.
.tail instructions Ignored OK Avoid a style that is dependent on optimization of complex tail calls.
F# Events OK Fails Use standard .Net Events on CoreRT.
Highly nested generics OK Fails (Not yet encountered in realistic code.)
char enums Fails ? Use int enums

Of the issues affecting UWP, lack of support for .tail does not break typical F# code (as Fable and historically mono AOT prove), and the other two issues have simple workarounds.

See https://github.com/FoggyFinder/FSharpCoreRtTest/blob/master/README.md#description-of-current-issues for more detail, and the MainIssues branch for relevant code.

.Net Native: gatekeeper.xml blocks F#

The remaining problem is a non-technical one. F# usage of .Net Native is blocked by gatekeeper.xml.

<FSharpRule on="true">
<ErrorLevel>error</ErrorLevel>
...
</FSharpRule>

This can be disabled on user systems to enable compilation with .Net Native, allowing creation of working UWP apps. However these apps cannot be successfully uploaded to the store since they will fail the F# test when compiling on Microsoft servers. We therefore need official editing of GatekeeperConfig.xml, either by turning it off or by replacing error with warning.

Reasons to allow F# UWP applications on the Windows Store:

  1. Already promised. 2015-10 @crutkas: "We are committed to bringing F# support to .NET Native."

  2. No technical work required. There are no technical issues currently blocking F# usage.

  3. Large user demand. 1500+ votes for Add F# support for .NET Native and 2000+ votes for F# support in .Net native for UWP when those were active. This thread is a continuation of F# Support in .NET Native and UWP and .Net Native F# Support; the latter contains use cases.

@morganbr
Copy link
Contributor

@morganbr morganbr commented Jul 11, 2018

@charlesroddie, thank you for investigating this in depth. Can you please tell us more about what goes wrong when you try to use sprintf, ToString and Quotations on .NET Native (UWP)?

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Jul 11, 2018

sprintf gives a System.IndexOutOfRangeException on UWP, as does using ToString on DUs and record types. On CoreRT there is an NRE. @zpodlovics identified the sprintf issue as coming from dynamic code generation:

let mi = typeof<ObjectPrinter>.GetMethod("GenericToString", NonPublicStatics)
let mi = mi.MakeGenericMethod(ty)
mi.Invoke(null, [| box spec |])

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Jul 13, 2018

There is no general issue with F# quotations; the only issue we have found is with LINQ. The function causing the problem is Microsoft.FSharp.Linq.RuntimeHelpers.LeafExpressionConverter.QuotationToExpression. This gives a fail fast error.

E.g. if db is an sqlite connection, db.QueryAsync<T>() .Table<T>() .OrderByDescending(fun x -> x.Timestamp) fails. A workaround is to use SQL: db.QueryAsync<T>("select * from T order by Timestamp desc").

LINQ requires care even from C# and inherently uses a lot of reflection I suppose.

F# query expressions seem to work OK.

@jptarqu
Copy link

@jptarqu jptarqu commented Aug 7, 2018

Has there been any progress on this (official editing of GatekeeperConfig.xml)?

Thank you all for all your hard work.

@MichalStrehovsky
Copy link
Member

@MichalStrehovsky MichalStrehovsky commented Sep 13, 2018

We are looking into removing the F# blocking in the Store compilation machines for the next release of .NET Native for UWP apps.

F# will remain officially unsupported, but we'll not prevent people who understand the implications (and who did enough testing to make sure it works okay for their use case) from uploading to the Store. People will still need to hack their local Visual Studio installation to remove the blocking locally because we don't believe the experience of using F# with the level of support the .NET Native compiler has is anywhere near pleasant.

The focus of the .NET Native team has been on completing ARM64 support in the next release. The work to lift the remaining technical limitations to fully support F# is unfortunately non-trivial.

For an example of the technical limitations on top of the issues found by the testing in the top post, you can try this:

  1. Create a blank UWP app project in Visual Studio
  2. Add a new F# .NET Standard 2.0 library project
  3. Put this in the F# project:
namespace FSharpLibrary

module Fib =
    let rec fib = function
        | n when n=0I -> 0I
        | n when n=1I -> 1I
        | n -> fib(n - 1I) + fib(n - 2I)
  1. Add a reference to the F# project from the UWP app and add a call to the method in F# from it.
  2. Build a Release version of the app

The .NET Native compiler will emit a couple warnings about unsupported LDTOKEN instructions to stdout (there's several places in FSharp.Core.dll where the IL is doing LDTOKEN of an uninstantiated generic method, such as in static System.Void <StartupCode$FSharp-Core>.$Query..cctor() - this is not currently supported by the native code generator backend we use. Then the compiler will crash with an internal error.

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Sep 13, 2018

Great news! Allowed but not officially supported is the sensible decision.

For an example of the technical limitations on top of the issues found by the testing in the top post...

Thanks. We will check this and keep the issue list updated.

@dsyme
Copy link

@dsyme dsyme commented Sep 13, 2018

The .NET Native compiler will emit a couple warnings about unsupported LDTOKEN instructions to stdout (there's several places in FSharp.Core.dll where the IL is doing LDTOKEN of an uninstantiated generic method, such as in static System.Void <StartupCode$FSharp-Core>.$Query..cctor() - this is not currently supported by the native code generator backend we use. Then the compiler will crash with an internal error.

We can likely fix this easily in a future FSharp.Core (e.g. change to emit ldtoken of an instantiated generic method and then call GetGenericMethodDefinition() )

@FoggyFinder
Copy link

@FoggyFinder FoggyFinder commented Sep 13, 2018

For an example of the technical limitations on top of the issues found by the testing in the top post, you can try this:

hm, I can't reproduce it:

test_uwp

Did I do something wrong?

@MichalStrehovsky
Copy link
Member

@MichalStrehovsky MichalStrehovsky commented Sep 14, 2018

We can likely fix this easily in a future FSharp.Core (e.g. change to emit ldtoken of an instantiated generic method and then call GetGenericMethodDefinition() )

That would help! We've had an issue open to make it work for a long time. It never met the bar because we thought only IL obfuscators emit that kind of IL (and we have hack in place that checks if the subsequent instruction is a simple pop, which fixes the obfuscator...).

hm, I can't reproduce it

What version of the .NET Native compiler did you use? I used 2.1.8. Also, do did you modify the RD.XML in the empty UWP app template? The problem goes away after removing the *Application* line because we no longer need to compile all of FSharp.Core, only a subset that's statically needed.

@FoggyFinder
Copy link

@FoggyFinder FoggyFinder commented Sep 14, 2018

What version of the .NET Native compiler did you use? I used 2.1.8.

The same.

Also, do did you modify the RD.XML in the empty UWP app template?

No, I didn't. My rd (without comments and whitespaces):

<Directives xmlns="http://schemas.microsoft.com/netfx/2013/01/metadata">
  <Application>
    <Assembly Name="*Application*" Dynamic="Required All" />
  </Application>
</Directives>

Version of FSharp.Core: 4.5.2

@MichalStrehovsky
Copy link
Member

@MichalStrehovsky MichalStrehovsky commented Sep 14, 2018

Here's my project: FSharpApp.zip

One thing I noticed is that the compiler only crashes when building x86. It compiles fine for x64.

@FoggyFinder
Copy link

@FoggyFinder FoggyFinder commented Sep 14, 2018

One thing I noticed is that the compiler only crashes when building x86. It compiles fine for x64.

yep, exactly, I had x64

@JaggerJo
Copy link

@JaggerJo JaggerJo commented Nov 16, 2018

Are we stuck on "Allowed but not officially supported" for now or are there any plans towards "Officially supported" ?

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Dec 6, 2018

@JaggerJo: yes we are, as fully described in @MichalStrehovsky's post.

We can confirm that .Net Native compilation works now on Store compilation machines, and we have released our beta F# app on the store (link).

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Mar 13, 2019

Direct access to UWP UI classes from F# depends on either

  1. Creating an F# template for a UWP class library analogous to the current C#/VB templates, or
  2. Exposing Windows.Foundation.UniversalApiContract as NuGet package, which would allow creating a plain net core/standard class library and adding this package.

Until then F# can only be used 1. in parts of a solution that don't have to reference controls directly (i.e. models, viewmodels), or 2. indirectly via Xamarin.Forms.

@pauldorehill
Copy link

@pauldorehill pauldorehill commented Jun 6, 2019

@charlesroddie are there any workarounds to getting access to the UWP UIs (similar to the gatekeeper hack)? I'm using Fabulous and writing some custom UWP renderers, but as it stands I need to use a C# UWP Library for the views.

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Jun 9, 2019

@pauldorehill Yes using C# for the custom renderers is the right approach and will be for the next year or so.

WinUI 3.0, in whatever form that takes, is likely to have better F# support.

@TheFo2sh
Copy link

@TheFo2sh TheFo2sh commented Oct 3, 2019

ok for uwp , now i can compile the x64 and submit it to the store which is good but what about the Arm and arm64 ? i hope that MS can find a solution for it as both F# and UWP are part of the .NET ecosystem.

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Oct 3, 2019

@TheFo2sh Are you having a specific problem on ARM or are you just conjecturing that there may be a problem?

it

It's possible that you are making a mistake here. Compiled native code is not submitted to the store.

@knocte
Copy link

@knocte knocte commented Jan 29, 2020

This doesn't work for me. Trying to publish a C# app (which uses a lot of F# libs) fails to compile, with this error:

Severity	Code	Description	Project	File	Line	Suppression State
Error		ILT0005: 'C:\Users\knoct\.nuget\packages\microsoft.net.native.compiler\2.0.2\tools\x64\ilc\ilc.exe --gatekeeper @"C:\Users\knoct\Source\Repos\geewalletFRONTEND\src\GWallet.Frontend.XF.UWP\obj\x64\Release\ilc\intermediate\gkargs.rsp"' returned exit code 1	GWallet.Frontend.XF.UWP			

When googling this error, one can only find this stackoverflow answer (https://stackoverflow.com/a/49142932/544947) which just recommends to disable the Compile with .NET Native toolchain checkbox, but if you do this and try to submit to the appStore, you get the error:

Package acceptance validation error: This package wasn't built for submission to the Microsoft Store. Make sure you're uploading a Release build with the .NET Native tool chain enabled.

@MichalStrehovsky
Copy link
Member

@MichalStrehovsky MichalStrehovsky commented Jan 29, 2020

@knocte You have to hack your local installation to remove the blocking locally, as described here: #5780 (comment). We don't have a nicer way to do that because this makes it very clear that it's getting into unsupported territories.

See my post above:

F# will remain officially unsupported, but we'll not prevent people who understand the implications (and who did enough testing to make sure it works okay for their use case) from uploading to the Store. People will still need to hack their local Visual Studio installation to remove the blocking locally because we don't believe the experience of using F# with the level of support the .NET Native compiler has is anywhere near pleasant.

@TheFo2sh
Copy link

@TheFo2sh TheFo2sh commented Feb 11, 2020

@charlesroddie , sure you can submit a package with f# to the store after turning off the .net native compiler guard , but Unfortunately this works only for the x64 bit. so now after Microsoft released the arm based surface pro x i need a solution to create an armx64 package that can be submitted to the store while keep using f#.

knocte added a commit to nblockchain/geewallet that referenced this issue Feb 19, 2020
When trying the option "(right click on project->)Publish->Create App
Packages..." I was getting the error:

ILT0005: 'C:\Users\knoct\.nuget\packages\microsoft.net.native.compiler\2.0.2\tools\x64\ilc\ilc.exe --gatekeeper @"C:\Users\knoct\Source\Repos\geewalletFRONTEND\src\GWallet.Frontend.XF.UWP\obj\x64\Release\ilc\intermediate\gkargs.rsp"' returned exit code 1	GWallet.Frontend.XF.UWP

Which seemed to be a limiation of UWP wrt F#, but after bringing it up
in github[1] I was recommended to follow the recommendation of hacking
my local environment[2]. I did this in the recommended path:

C:\Program Files (x86)\Microsoft SDKs\UWPNuGetPackages\runtime.win10-x64.microsoft.net.native.compiler\2.2.7-rel-27913-00\tools\x64\ilc\tools\GatekeeperConfig.xml

But it didn't have any effect. So I did it in this path instead, which
seems more appropriate by reading the initial compilation error I had:

C:\Users\<MyUserName>\.nuget\packages\microsoft.net.native.compiler\2.0.2\tools\x64\ilc\tools\GatekeeperConfig.xml

But the result of doing this was this strange error:

Internal compiler error: Object reference not set to an instance of an object.

So I decided to upgrade the nuget packages of the project related to the
UWP SDK (from 6.0.4 to 6.2.9 as this diff shows), and it seems to have
helped here because I don't get this error anymore (but others which I'll
fix in subsequent commits).

[1] dotnet/corert#6055 (comment)
[2] dotnet/corert#5780 (comment)
@knocte
Copy link

@knocte knocte commented Feb 22, 2020

@MichalStrehovsky I appreciate your comment. However I have tested what is advised in that issue/comment, and while some compilation errors go away, very basic F# functionality such as sprintf and printf simply throws NullReferenceExceptions. Where can I submit this bug?

@charlesroddie
Copy link
Contributor Author

@charlesroddie charlesroddie commented Feb 22, 2020

@TheFo2sh If you'd like help I'd suggest posting a precise description of the successful process with just C# and at what point an identical process with F# code fails.

@knocte The current way round is to remove sprintf and printf. If you want to use sprintf in reflection-free/unfriendly toolchains, or anywhere where performance is important, then the best places to raise an issue are https://github.com/dotnet/fsharp or https://github.com/fsharp/fslang-suggestions to discuss reflection-free string formatting. Or add to dotnet/fsharp#4954 .

@knocte
Copy link

@knocte knocte commented Feb 23, 2020

@charlesroddie thanks, I didn't know it was because of reflection. I have a workaround in mind, I will test it and will report about my findings after that, in case I see more limitations.

@AngelMunoz
Copy link

@AngelMunoz AngelMunoz commented Feb 25, 2020

Hey there not sure if this is the right issue (if not please point me to the correct one) but I have an issue trying to use Microsoft.Windows.SDK.Contracts that as far as I know, these are WinRT API's
here's a reproduction repository
https://github.com/AngelMunoz/ContractsRepo

knocte added a commit to nblockchain/geewallet that referenced this issue Mar 17, 2020
It turns out that F#'s (s)printf(n) functions use reflection [1],
so UWP's CoreRT compiler/runtime is not happy about it at all [2];
then this is a workaround that uses `String.Format` underneath when
compiled with the REFLECTIONLESS define. (Not compiling with this
define will still give us the compile-time safety of sprintf vs
String.Format such as checking number of arguments and their types).

[1] dotnet/corert#6055 (comment)
[2] https://stackoverflow.com/q/60350735/544947
knocte added a commit to nblockchain/geewallet that referenced this issue Mar 17, 2020
It turns out that F#'s (s)printf(n) functions use reflection [1],
so UWP's CoreRT compiler/runtime is not happy about it at all [2];
then this is a workaround that uses `String.Format` underneath,
except when compiled with the a define for stricter compilation.
(Compiling with this define will still give us the compile-time
safety of sprintf vs String.Format such as checking number of
arguments and their types, but without the portability to UWP.)

[1] dotnet/corert#6055 (comment)
[2] https://stackoverflow.com/q/60350735/544947
knocte added a commit to nblockchain/geewallet that referenced this issue Mar 17, 2020
It turns out that F#'s (s)printf(n) functions use reflection [1],
so UWP's CoreRT compiler/runtime is not happy about it at all [2];
then this is a workaround that uses `String.Format` underneath,
except when compiled with the a define for stricter compilation.
(Compiling with this define will still give us the compile-time
safety of sprintf vs String.Format such as checking number of
arguments and their types, but without the portability to UWP.)

[1] dotnet/corert#6055 (comment)
[2] https://stackoverflow.com/q/60350735/544947
knocte added a commit to nblockchain/geewallet that referenced this issue Mar 17, 2020
It turns out that F#'s (s)printf(n) functions use reflection [1],
so UWP's CoreRT compiler/runtime is not happy about it at all [2];
then this is a workaround that uses `String.Format` underneath,
except when compiled with a define for stricter compilation.
(Compiling with this define will still give us the compile-time
safety of sprintf vs String.Format such as checking number of
arguments and their types, but without the portability to UWP.)

[1] dotnet/corert#6055 (comment)
[2] https://stackoverflow.com/q/60350735/544947
knocte added a commit to nblockchain/geewallet that referenced this issue Mar 18, 2020
It turns out that F#'s (s)printf(n) functions use reflection [1],
so UWP's CoreRT compiler/runtime is not happy about it at all [2];
then this is a workaround that uses `String.Format` underneath,
except when compiled with a define for stricter compilation.
(Compiling with this define will still give us the compile-time
safety of sprintf vs String.Format such as checking number of
arguments and their types, but without the portability to UWP.)

[1] dotnet/corert#6055 (comment)
[2] https://stackoverflow.com/q/60350735/544947
knocte added a commit to nblockchain/geewallet that referenced this issue Mar 18, 2020
It turns out that F#'s (s)printf(n) functions use reflection [1],
so UWP's CoreRT compiler/runtime is not happy about it at all [2];
then this is a workaround that uses `String.Format` underneath,
except when compiled with a define for stricter compilation.
(Compiling with this define will still give us the compile-time
safety of sprintf vs String.Format such as checking number of
arguments and their types, but without the portability to UWP.)

[1] dotnet/corert#6055 (comment)
[2] https://stackoverflow.com/q/60350735/544947
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet