Support CoreCLR

[lstratman]

Are there any plans to support using CoreCLR as the .NET runtime under Linux to provide an alternative to Mono? I've read in a few places that CoreCLR provides a .NET hosting API which is what Kestrel apparently uses to provide ASP.NET hosting for the ASP.NET vNext project under Linux. If there's nothing on the roadmap, I may fork and try to implement this myself, but I figured that I'd put something out there to see if anyone else was already working on this functionality.

[tjanczuk]

Support for CoreCLR would be a very welcome addition to Edge.js. I don't have the cycles at the moment to drive it myself, but if you decide to invest in this I will be happy to participate and take a PR.

[lstratman]

Thanks for the quick response! I'll fork and see what I can do...

[rscharnetzki]

I agree, support for CoreCLR would be a very logical step for Edge.js!

[lstratman]

For those of you interested in this functionality, I have a fully functional CoreCLR implementation of Edge.js up at https://github.com/medicomp/edge. All unit tests pass and performance is right up there with the existing Mono implementation (0.16 seconds on the performance tests on my VM under CoreCLR vs. 0.14 seconds for Mono). I've updated the setup scripts in my fork to install CoreCLR as part of the setup process for anyone interested in checking it out; see https://github.com/medicomp/edge#building-on-linux for instructions.

A few things to note:

• Only Linux support so far. I'll be working on that shortly (nothing difficult about it), but I figured that Linux would be a good place to start with considering that Windows obviously already has an official, Microsoft-supported CLR :).
• Mono is still required. Microsoft hasn't released the Roslyn C# compiler bits for CoreCLR yet, so we still need Mono to compile the Edge.js .NET assemblies. A CoreCLR compiler is forthcoming from Microsoft and I will update binding.gyp accordingly once it's released.
• Since we don't have a compiler yet for CoreCLR, compiling C# or VB.NET from JavaScript strings or function comments doesn't work. Again, once the compiler is released, this will be rectified.
• The .NET assembly references the CoreCLR runtime assemblies directly during compilation instead of contract assemblies from NuGet. It works, but it's not optimal and has the potential to limit portability if you're jumping between a bunch of different versions of the CLR; if you do that, I'd recommend rebuilding Edge.js when you do so. The dnu command to restore packages from NuGet doesn't work right under CoreCLR on Linux in the current beta, and that's the way I'd like to get the contract assemblies for use during compilation. Once dnu is stable and functional in Linux on CoreCLR, I'll update binding.gyp to do the .NET assembly compilation "the right way".

Since Mono is currently necessary, the compiled version of Edge.js will include both Mono AND CoreCLR support. In this dual-runtime case, I elected to use Mono by default for compatibility purposes, so if you want to use CoreCLR, just specify the EDGE_USE_CORECLR=1 environment variable when starting Node.js.

That about covers my brain dump for now. For those that are interested, please check it out, post comments or questions, give me PRs, whatever. Once the above gotchas are resolved, I'll issue my own PR or I'm happy to do so sooner if you want it.

[tjanczuk]

This looks great @lstratman!

What are the benefits of using CoreCLR instead of Mono right now (no code compilation)? What will be the benefits once the C# compiler is supported by CoreCLR itself?

Did you not do OSX support for lack of time or is there something more fundamental?

[lstratman]

IMO, the two big advantages (once CoreCLR is released) will be the intangibles: consistency in the CLR between various OSes and ongoing support from a big corporation in the form of Microsoft. My company currently uses Edge.js as an option to host our .NET web services within client Node.js applications on both Windows and Linux and the differences between Mono and Windows .NET mean we have a lot of #if directives in our code and a lot of stress and time goes into ensuring consistent behavior in .NET between the various OSes. Given the fact that the JITer, the garbage collector, the base classes, etc. will be the same across all OSes is a huge stability win. Then you get the fact that this is all developed and supported by Microsoft and the fact that non-Windows CLRs now have performance enhancements like RyuJIT; the clients that I've dealt with are much more comfortable with a Microsoft-developed and -supported CLR on Linux.

Once CoreCLR gets the C# compiler, the compilation under CoreCLR may be a little faster than under Mono, although that's pure conjecture on my part, and you'll have the benefit of consistent IL code being generated across both Windows and *NIX/OS X. Functionality-wise, though, the Mono and CoreCLR-compiled code will be pretty much identical: whatever you can compile in Windows .NET, you can do on Mono. I should also be able to get the actual execution performance under CoreCLR to meet or exceed that under Mono once I get things working on Windows and can bring some profiling tools to bear.

Thanks for reminding me about OS X, that's also on my TODO list. CoreCLR is fully supported on OS X, so getting Edge.js to support it as well should be relatively straightforward.

[tjanczuk]

This makes sense.

Do you think CoreCLR can be considered as a replacement of .NET on Windows in the context of Edge.js usage? I wonder if Edge can normalize on one CLR implementation across platforms.

I will be happy to take this as a PR, but without code compilation it strikes me as of limited use. Do you think we should work on the PR now or wait until CoreCLR gets compiler support?

Lastly, did you test it against all the supported node versions in both 86 and 64 bit versions, or just the latest?

[lstratman]

I completely agree: I would definitely wait until compiler support comes around. Also gives me a chance to get performance to where it needs to be :).

I think that, on Windows at least, there will still need to be support for the existing CLR (UncoreCLR?): given the amount of work that needs to go into porting some applications over to CoreCLR, I imagine there are many projects where that's either way down the road or not even a possibility. On Linux and OS X, there's a much greater possibility of dropping Mono support since I imagine that Linux users of Mono are chomping at the bit (as I am) to move to CoreCLR. But that might not be necessary. For the CoreCLR stuff, I do most of the marshalling of data between .NET and V8 either in C# or in standard C++. I don't have any real dependencies on C++ data structures from the CLR and on the C# side, it's all standard calls to the Marshal class that Mono also supports. Theoretically, this code (both the C# and C++) should be able to run under Mono with only some #ifdefs necessary to load up the Mono runtime and get a few delegates for the purposes of loading MonoEmbedding.exe. It allows you to continue to support Mono but from a much more unified codebase with a minimum of Mono-specific code.

It's all a big if right now, but it might be worth looking into.

As to your last point, so far I've done the vast majority of my testing in Node.js 0.12, but will also test this under 0.10. Are there other Node.js versions that you're interested in supporting?

[tjanczuk]

Sounds good. Let me know when you think this is ready to PR, I am looking forward to it.

0.12 and 0.10 are fine. By the time CLR gets the compiler we may only need 0.12 (or hopefully 1.0)

[lstratman]

OK, it's been a while, but my CoreCLR branch is in a good state for folks to start testing with: performance is where it needs to be, all tests pass across Windows, Linux, and OS X (there appears to be an issue with the latest CoreCLR builds for OS X, but I'll look into that) on the latest CoreCLR build and Node.js 0.8 through 4.1.1, and C# dynamic compilation works. One item of note is that edge.node is now edge_nativeclr.node and edge_coreclr.node. This was primarily a consequence of Windows compatibility since the Node.js module shouldn't contain any native CLR bootstrapping code (the /clr build switch specified during compilation), so I had to compile two versions of the module.

As I said, I think the code is feature complete and is in as good a shape as any for initial testing. I updated the README as extensively as I could and will be issuing PRs for both the edge and edge-cs repositories shortly. If you run into problems, please don't hesitate to let me know.

[lstratman]

Actually, it's probably a good idea to put the PR in a branch, at least initially. Once you have a branch in your edge and edge-cs repositories setup, let me know and I'll send.

[tjanczuk]

@lstratman This is great news, thank you for all the hard work. I created coreclr branches in edge and edge-cs.

C# dynamic compilation works

How did you make this work? Has something changed in the CoreCLR ecosystem to enable it since we last spoke?

[lstratman]

How did you make this work? Has something changed in the CoreCLR ecosystem to enable it since we last spoke?

I'm basically using the same approach that Microsoft themselves use for dynamically compiling and running ASP.NET 5 applications when you do a dnx run from within an ASP.NET project directory. They ship the Roslyn Microsoft.CodeAnalysis.dll and Microsoft.CodeAnalysis.CSharp.dll assemblies as part of the distribution package, so I can just leverage those to do in-memory compilation much the same way that the existing dynamic compilation works.

The sticky part is bringing in references. You can manually specify reference assembly paths the same way that you do now via #r directives in the code itself or the arguments to edge.func(). However, you would also have to specify all of the descendant references since the CoreCLR assembly loader doesn't do things like look in bin directories, the current directory, etc. for those descendants. The "new" way of doing things for CoreCLR is to use a project.json file and, by extension, NuGet to specify those dependencies and then run dnu restore before running your application to download all of the necessary packages and generate a full package dependency tree that's saved off as project.lock.json. So, Edge.js users can create a project.json file specifying all of the reference packages for their dynamically-compiled code and then run dnu restore to bring all of the packages down. When running under CoreCLR, I check the application path during startup (either the current directory or whatever the user specified in the EDGE_APP_ROOT environment variable) for a project.lock.json file and, if it's present, I parse it and load up all of the package assemblies and their dependencies. Then, when a package is referenced from dynamic code via #r System.Data.SqlClient.dll, I can get the assembly for the System.Data.SqlClient package from what was downloaded for the project.json file. It's pretty much the same way that ASP.NET 5 works.

[tjanczuk]

@lstratman very smart, thanks for the update.

[jskulavik]

Thanks for this @lstratman! This is fantastic. Would you and @tjanczuk think this is stable enough to merge the coreclr branch into master? The RC bits for CoreCLR are starting to show up in NuGet/MyGet and it'd be nice too have edge.js ready to support it right as it hits RC in a couple of weeks

[tjanczuk]

Yes this was fantastic work. I was heads down in a parallel universe for the last 2 weeks or so, but I hope to go over this PR and merge in during the next week.

[pksorensen]

If anyone has updates from other discussions about if we will be able to run NodeJS stuff from CoreCLR in any future, an update here would be nice :) I would help but no idea where to start.