Add .NET Core (CoreCLR) build of FSharp.Compiler.Service.dll

[abrodersen]

The new RC of CoreCLR, CoreFX, and dnx means that applications can now be built using these tools. Please add a Nuget package targeting dnxcore or better yet netstandard so that F# apps can be written on this new platform. The CoreFX team has a great writeup on the new standard dotnet API. A compiler plugin for dnx has already been written.

[dsyme]

Just to check - do you want

1. F# support on CoreCLR/CoreFX (which is already in progress in the coreclr branch of Microsoft/visualfsharp), or
2. an FSharp.Compiler.Service package that targets netstandard (also a good idea, but a different request)

If 2 please add more details about your use cases.

[abrodersen]

Essentially, I would like the ability to compile and run F# code on CoreCLR/CoreFX using the new project.json based build system. Right now, AFAIK, the dnx utility is the only way to do so, and it requires the aforementioned compiler plugin to compile F# code. This plugin depends on the FSharp.Compiler.Service package to function. The plugin currently only targets dnx451 a.k.a Desktop CLR because FSharp.Compiler.Service does not target dnxcore or netstandard.

Are you planning to add official support for building F# code using the new project.json system? If so, then request (2) is not needed.

[dsyme]

Ah ok - I see.

We'll need to integrate the work from Microsoft\visualfsharp coreclr branch once it's stable - those are the changes needed to bring up the compiler (and this compiler service) on CoreCLR. I think the core of the work relevant to the compiler service is already stable and can be brought over - perhaps into a CoreCLR branch here.

[abrodersen]

Sounds great! I'll keep an eye out for said branch :)

[dsyme]

OK, it's time to bring this to life again...

The steps we need to get a .NETCore FCS are:

:[x]: Integrate from Microsoft/visualfsharp --> fsharp/fsharp --> fsharp/FSharp.Compiler.Service

:[ ] Add build support for CoreCLR to this repo. Basically copy the appropriate dependencies and settings from Microsoft/visualfsharp to this repo if they didn't come across as part of the integration.

:[ ] Build it, test it on CoreCLR

:[ ] Ship an updated Nuget

For #1, there is an integration from Microsoft/visualfsharp --> fsharp/fsharp going on here; fsharp/fsharp#558. It's still work in progress

[abrodersen]

Thanks for the update! The F# tooling for .NET Core has mostly fulfilled my requirements that originally motivated this issue. However, I'm sure there will be many other potential use cases for a .NET Core FCS package for other developers. Regardless, thanks for the effort to bring F# into the Core fold!

[dsyme]

@ncave The first step is now complete. Would you be able to help add a .NET Core build of this component?

[ncave]

@dsyme Awesome! Just to clarify again, is your preference to start here, or can we add coreclr FCS to fsharp/fsharp, as stated in the Looking Ahead section, paragraph 1 of fsharp-contributions?

[dsyme]

@ncave We should start here. At least, give it a go and find out why it's hard. The code is all ready and integrated after this, we just need to work out how to build and test it. It can't be that hard, can it? :)

One of the reasons I'd like to do things here is to understand better what it means to add a .NETCore build+test for an existing, standard F# repository. This is a somewhat orthogonal goal but we need to make sure it is "easy". We are going to have to do this for 100s of repositories over the coming year.

There are various ways to add .NET Core support to a repo. Some examples are:

• FsSrGen, this a little executable tool. The repo builds and runs both a .NET Framework and a .NET Core version of the tool, so in theory it's a lot like what we have to do here. It uses both a .fsproj and a project.json, sort of a parallel universe of two build systems. The .NET Core support is complex and looks very fragile - there are Powershell scripts, which I'd very much like to avoid, there's a gnarly Travis config file to run .NET Core code from CI on Linux, and likewise a gnarly AppVeyor config file. While it's great to have this as an examplar to prove that it's actually possible, it also indicates just how "raw" end-to-end build+test of .NET Core support still is (and remember, this is just to build+test a very simple cross-platform .EXE). I'd expect this stuff to simplify massively in the next few months. Much of it has nothing to do with F#.
• Visual F# This repo also builds (and runs) a .NET Core version of the F# compiler itself. The build system is complex, but you successfully augmented the .NET Core build of FSharp.Compiler.LanguageService to do the right thing.
• Suave. This is another good exemplar, it does a build+test. But again the contortions in the appveyor.yml concern me, and the travis.yml has not been updated. Paket is not used for .NET Core package management (project.json + dotnet CLI is used instead), so the repo is left in a "fractured" or "two-parallel-build-systems" state. The repo uses both a project.json and a Suave.fsproj
• The F# .NET CLI samples , linked by @enricosada below

All this is kind of unsatisfactory: to be honest it seems that adding .NET Core support is still quite hard. And I will admit that I hate fractured, parallel-world build systems. I can see now why you just went and modified Microsoft/visualfsharp (because the build+test is already set up in that repo). But equally we must make adding .NET Core build+test support simpler.

I was also curious to know if it's possible to use Paket+.fsproj+msbuild to build dnxcore/netcore/netstandard components (though all the stuff linked above will still presumably be needed if we want to test anything). What I found is this:

• A suitable addition to paket.dependencies appears to be this:

source https://www.myget.org/F/dotnet-core/api/v3/index.json
source https://api.nuget.org/v3/index.json
source https://www.nuget.org/api/v2/

nuget NETStandard.Library 

• However, Paket doesn't yet support the "netstandard" monikers that came out in the last few days. Support is being added.. I'm not sure there's any workaround short of winding back to earlier reference assemblies which used the "dnxcore" monikers.
• We still need an FSharp.Core for .NET Core. @enricosada says this is in the feed source https://www.myget.org/F/fsharp-daily/api/v3/, browse here, though there is also something available here. I'm not totally keen on taking dependencies "daily" feeds but the status of the second thing is not clear either. In either seems best to add a netstandard version to the normal FSharp.Core nuget package, so I've added an issue for that here.
• In general, Paket has a lot of problems processing .NET Core dependency listings. For example, I made a paket.dependencies from the dependency listing from the project.json in your checkin. However paket didn't successfully resolve this - after 8 minutes I had to stop it. I've added this as a Paket issue here

[enricosada]

I'd like too to see a clean implementation of a fsproj supporting .net core, with paket if possibile.
So if i can help, pls tell me.

Long comments, as usual sorry

Some more info:

fsharp daily is not daily ci

The nuget it's published when a new version is needed.
FSharp.Core .NET Core nuget is:

• name: Microsoft.FSharp.Core.netcore
• version: 1.0.0-alpha-160412
• feed: https://www.myget.org/F/fsharp-daily
• supported frameworks: netstandard1.5

the previous version supported dnxcore50 only, not netstandard.
dnxcore50 is deprecated.
docs: http://dotnet.github.io/docs/libraries/libraries-with-cli.html and https://github.com/dotnet/corefx/blob/master/Documentation/architecture/net-platform-standard.md

About put netcore in that in same FSharp.Core nuget, pls do it.
But it's i think there are issues about alignment/sync/reference, but i'll add a comment to the issue
Sry no changelog

About .net cli

To write info about current situation (initial committed code may be old/dirty)
It's not to defend project.json based, but i think it has a lot of features the msbuild lack (first class support for packages, easy to read, easy multi framework support).
And .net cli enhance project.json based (it's only a config file) with stupid simple commands.
All together, i dont want to touch msbuild files anymore, because i can do the same job with project.json + .net cli. And it's oss

build script

to run a build and create a package, you need only to do

dotnet restore
cd projectDir
dotnet pack

dotnet pack already call dotnet build to compile for every framework specified in project.json, ootb, with nuspec attributes like authors, tags etc writte in the project.json.

to build and run a project (like tests)

dotnet restore
cd projectDir
dotnet run -- argument1 argument 2

appveyor and travis

both config just download a script. The script download the .net cli ZIP, unzip in a subdirectory, and add that to PATH. nothing fancy.

a clean template/example of both are:

• https://github.com/enricosada/fsharp-dotnet-cli-samples/blob/master/appveyor.yml
• https://github.com/enricosada/fsharp-dotnet-cli-samples/blob/master/.travis.yml

i can do curl | bash with one liner, i think like that it's easier to read. but maybe less lines is better.
Current it's like when mono was installed before travis supported that ootb.
work is in progress in travis csharp image to add a -dotnet: version like -mono: version after dotnet cli rc2 is released.

also it's possibile to use system packages instead of unzip in a directory, but i prefer unzip because i can try that locally without messing up my machine.

official docs of .net cli install scenario: https://github.com/dotnet/cli/blob/rel/1.0.0/Documentation/cli-installation-scenarios.md

my build scripts in FsSrGen and other pr

I write a lot of code for build because i like to split commands in multiple task/targets.
It's not needed, as i said, just restore + pack are needed.
I used powershell because i tried experience without dependencies on build tools.

In the other pr (chessie/suave) i tried initially to NOT mess with their build file, and run commands in appveyor/travis after build.
The updated pr now integrate in fake/rake, and run commands as fake/rake tasks. And these task can be merge in a single one.

Some examples:

• in a fake target, like fsx in chessie pr running cmd like Shell.Exec("dotnet", "restore")
• in a rake task, like Rakefile in suave pr running cmd like system "dotnet restore"
• from script, like powershell or bat or bash or appveyor, just running cmd like dotnet restore

[ncave]

Thanks @dsyme and @enricosada, the examples help a lot.

[dsyme]

@ncave Overall, I'd say that for now we would accept a PR that adds .NETCore build + test support along the lines of Suave or FsSrGen, even if that means we end up with two build systems and duplication of the messy stuff from those repos. Presumably we can clean it out later.

Criteria would be

• Add a suitable project.json, plus all the ugly appveyor.yml, travis.yml stuff .... No use of Paket.
• Add it to the nuget package
• Run tests

Perhaps much of that travis/appveyor/powershell stuff could go away if some CoreCLR helpers were added to FAKE.

I'm actually les concerned about running tests than you might think and we could release a pre-release without it, as long as a basic manual test has been done. This code is a fork of http://gitub.com/Microsoft/visualfsharp, we can assume CoreCLR testing has been done.

[dsyme]

@enricosada Thanks for the notes!

dotnet cli is nice as a project builder. Is there FAKE support for it yet? Does there need to be? That would seem to be important. Perhaps it's a one liner but it would be nice to have a sample.

Currently I feel that the simple task of targeting "netstandard" as a profile is forcing people to change everything:

• framework: fragmented netstandard instead of unified .NET 4.6,2
• project build system : dotnet-cli instead of MSBuild
• dependency manager : nuget v3 instead of Paket
• editor tooling : VS .xproj instead of fsproj
• build automation : PS scripts instead of FAKE helpers

This amount of change is really hard to get your head around.

I just wish things were a bit more independent, e.g.

• project build system: "You can use either dotnet-cli or MSBuild to create either netstandard or netfx components."
• dependency manager: "You can use either project.json or Paket to declare and manage dependencies"
• editor tooling: "You can use either xproj+dotnet-cli or fsproj+MSBuild when targeting either netstandard or netfx"
• build automation: "You can use FAKE with either xproj+project.json+dotnet-cli or fsproj+MSBuild"

[enricosada]

@dsyme i can do the pr project.json, if you want assign this issue to me
I have already done that for visualfsharp ( fsharp.core/fsc/fsi/fsharp.compiler ) so i know issues (for
example we need fssrgen from package)

test it's easy, we can run nunit 3 tests (or whaterver) with .net core, already done that for chessie, ref this commit

[enricosada]

build automation: "You can use FAKE with either xproj+project.json+dotnet-cli or fsproj+MSBuild"

Yes, @dsyme see my previous comment, there is the example of fake integration in chessie.
It's just Shell.Exec("dotnet", "restore") in Fake

[dsyme]

@enricosada I think it's important to make this replicable, so other people can do what you've been doing, and are happy to do it. So I might ask you to let @ncave have a crack at this (@ncave - what do you think - would you like to?), and then review?

Some other comments

• I'd think I'd really like to see all the Powershell/Bash stuff in travis/appveyor.yml become FAKE helpers. Or somehow factored out. I definitely want to keep FAKE as the overall build automation.
• I'd like to limit the role of "project.json" to just a project build, giving a single DLL+XML+PDB (rather than producing an intermediate nuget package just for the netstandard DLL which we then have to merge)
• No powershell please unless entirely unavoidable

[dsyme]

@enricosada Thanks for the FAKE sample, that's helpful.

Is it possible to avoid the dependency on "mergenupkg" here? I don't want to depend on any non-standard extensions to dotnet-cli, and simply use it as a DLL builder called by FAKE.

[enricosada]

@dsyme non standard extension is a dotnet cli tool ( inside tools in project json, ref doc ).
It's a normal package with a normal console app built for netstandard1.5. you can download the package, and dotnet run it
If you put that package foo inside the property tools in project.json, the dotnet restore download also that package, and you can run it with dotnet foo argument1 argument2. RAD.

Atm is not possible, but the only problem i see is paket template (for create the nuspec) doesnt support multiple frameworks ( ref fsprojects/Paket#1539 ). We can add files (it's easy with paket template), but not new dependencies for netstandard1.5
For that (and because not everyone use paket) i added dotnet mergenupkg, the packages are a zip files and nuspec is an xml so it's easy to move thing.

[enricosada]

@enricosada I think it's important to make this replicable, so other people can do what you've been doing, and are happy to do it. So I might ask you to let @ncave have a crack at this (@ncave - what do you think - would you like to?), and then review?

Sure, just ping me if someone need help. a basic migration guide is there.
https://github.com/enricosada/fsharp-dotnet-cli-samples/wiki/Migrate-fsproj-To-.NET-CLI
I used that for FsCheck, it's long because:

1. first create the project and use net46 framework so you fix all project.json issues (deps, files, f# 3 -> 4 sometimes)
2. add netstandard (.NET Core) and fix .net core issue (deprecated api, etc)

so you dont get all the issues together, more mechanical

[dsyme]

@dsyme .... is a dotnet cli tool ( inside tools in project json, ref doc ). It's a normal package ...

Yep, I know. But it's one more tool and I really want to Occam's Razor this.

For the purposes of what we're trying to achieve here, we don't need to do a dotnet publish, do we? We just want to dotnet build generating a DLL? I assume it is possible to just use dotnet as a project builder, without creating and merging these intermediate nuget packages? Thanks.

[enricosada]

@dsyme sure, just tell paket to download the dotnet-mergenupkg package, unzip, and dotnet "path/to/dotnet-mergenupkg.dll" argument1 argument2
the tool just remove that step, without paket it's annoying to use nuget directly

[dsyme]

Sure, just ping me if someone need help. a basic migration guide is there.
https://github.com/enricosada/fsharp-dotnet-cli-samples/wiki/Migrate-fsproj-To-.NET-CLI

Two things

• Could we move enricosada/fsharp-dotnet-cli-samples to fsprojects/fsharp-dotnet-cli-guide to make it canonical?
• Could we move that wiki to a README.md so it can accept PRs?

thanks!

[dsyme]

@dsyme sure, just tell paket to download the dotnet-mergenupkg package, unzip, and dotnet "path/to/dotnet-mergenupkg.dll" argument1 argument2

Not quite what I meant - I mean we just call dotnet build and mention the generated DLL in the existing package template, just like we do for existing DLLs. FAKE's packaging support does the rest.

[enricosada]

Could we move enricosada/fsharp-dotnet-cli-samples to fsprojects/fsharp-dotnet-cli-guide to make it canonical?

i think i am moving that to github.com/dotnet/netcorecli-fsc/ the new home dotnet-compile-fsc , the tool who call fsc after dotnet build.
I just need to finish some commit in that repo and create the official package, atm is inside dotnet/cli repo

Could we move that wiki to a README.md so it can accept PRs?

The wiki is public, i wrote about it the home, just change it, i'd love to (for example FAQ)
For example i got some help from @NumberByColors , to polish things

[dsyme]

i think i am moving that to github.com/dotnet/netcorecli-fsc/ the new home

OK, great

The wiki is public

OK, thanks!

[enricosada]

just like we do for existing DLLs. FAKE's packaging support does the rest

@dsyme fake packaging use paket template, and you can put additional file inside, but not specify dependencies for additional frameworks for nuspec, so i cannot add depdencies for netstandard1, see fsprojects/Paket#1539
Atm the nuspec generated by paket template doesnt specify the framework, that's the fallback, see https://docs.nuget.org/create/nuspec-reference
dotnet mergenupkg change the nuspec xml and add dependencies inside a <group target="NetStandard1.5">, and existing inside a fallbacl <group> (like before, same behaviour)

[dsyme]

@dsyme fake packaging use paket template, and you can put additional file inside, but not specify dependencies for additional frameworks for nuspec, so i cannot add depdencies for netstandard1 , see fsprojects/Paket#1539

Ah, I see! Yes, so that information in the intermediate nuget package is very valuable, I now understand why you ended up with dotnet-mergenupkg.

If Paket supported both "netstandard" and fsprojects/Paket#913 then I assume you wouldn't need this step?

[enricosada]

If Paket supported both "netstandard" and fsprojects/Paket#913 then I assume you wouldn't need this step?

@dsyme yes, should work.
If you use a nuspec file and nuget pack, for example, you dont need dotnet mergenupkg
I expect dotnet mergenupkg to be required for a short amount of time, it's just easy to use to fix an issue

[ncave]

@dsyme Thanks I'll take a crack at it with fake as in the Chessie netcore build that @enricosada contributed (great wiki, thanks!).

[dsyme]

@ncave Awesome. Don't hesitate to ask questions - we're all learning here :) I even grab @enricosada on Skype sometimes :)

[enricosada]

@ncave what's missing there? how can i help?

As recap, what's ok/missing?

• build using .NET Core SDK preview1
  • fw netstandard1.5
  • use only corefx deps of preview1, published on nuget.org (not dev feed)
  • use latest FSharp.Core netcore in fsharp-daily
• put built assemblies inside normal package ( using fake, paket or dotnet mergenupkg )

[enricosada]

i checked project.json, i think only deeps need to be changed

the .NET Core deps are merged inside normal FCS package?

i dont see where that's is done. i think is useless to have two packages (normal and .netcore), just one with multiple framework supported.
If is already like that ok, otherwise i can help (just use https://github.com/enricosada/dotnet-mergenupkg or paket)

deps

deps

  "dependencies": {
    "Microsoft.NETCore.App": {
      "type": "platform",
      "version": "1.0.0-rc2-3002702"
    },
    "System.Diagnostics.TraceSource": "4.0.0-rc2-*",
    "Microsoft.FSharp.Core.netcore": "1.0.0-alpha-*",
  },

use "NETStandard.Library": "1.5.0-rc2-24027" instead of Microsoft.NETCore.App, that's for console app.
you can see project.json.lock for useful deps for Microsoft.NETCore.App

also i think we can remove import of dnxcore50 from netstandard1.5, should work anyway because latest fsharp.core is netstandard, not dnxcore50 anymore. The others are corefx and are already netstandard

appveyor

Also appveyor doesnt need two build job anymore i think

[ncave]

@enricosada Thanks for the dotnet-mergenupkg link.

• No, the merging with the main FCS package is not done yet. I wholeheartedly agree the .netcore package needs to go away, so by all means if you can merge it, go ahead. It still has dependencies on the alpha versions of the Microsoft.FSharp.Core.netcore package, but I'm sure it will eventually be merged into the main FSharp.Core package too.
• The dependency on Microsoft.NETCore.App is a shortcut because FCS uses a lot of packages beyond NETStandard.Library. Of course, ideally we should not use either, but use trimming instead.
• Yes, if you can merge the nuget packages, the appveyor build jobs need to be merged too. Purely dotnet core TravisCI build is still TBD due to some xplat tools missing.

[enricosada]

@nave just open project.lock.json and copy deps of .NETCore.App (removing useless), that's it. It's a strange package, for apps. also you can see package needed in the project.json of the visualfsharp repo

I'll send a pr for preview2 fix and dotnetmergenupkg (it's already in dev, works ok, probably today)

i think i can just update the fslexyacc (current sources) to dnc, as dotnet-fslexyacc so we have xplat. @7sharp9 is working a lot on this, and need it, i'll do that.

[dsyme]

This has been added, a separate bug tracks nuget package for this