New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support building mscorlib on UNIX systems. #170

Closed
shahid-pk opened this Issue Feb 9, 2015 · 42 comments

Comments

Projects
None yet
10 participants
@shahid-pk
Contributor

shahid-pk commented Feb 9, 2015

Right now their is no way to build mscorlib on UNIX based systems.We should explore the options that we have to support building mscorlib on UNIX and adopt the best one.

some constraints

  • We will need mono's C# compiler at least in the best case, but if we go with creating visual studio projects using cmake we will have to use mono's xbuild too.
  • cmake does not support C#. Although it can create visual studio projects for C++ , supporting C# in those projects needs some hacking.
  • If we are to create visual studio project using cmake and hack it for C# we probably should be creating visual studio 2010 projects because mono's xbuild does not seem to play nice with visual studio 2013 projects.

Suggestions
The best way that i think of would be to create cmake modules for C# support, some people and projects have done it already and it is not that difficult either
for example https://github.com/awakecoding/CMakeSharp
Mono projects also use their C# extensions for the make build system. But those are for make.

This solution has some advantages

  • we will only need mono's mcs which is almost feature complete and avoid xbuild which is very buggy.
  • we will be able to replace mono's mcs with roslyn if in future it became available on UNIX and completely remove mono dependency from the UNIX build but that is a long term solution of course.

Update
Msbuild has been open sourced and it is going to be availabe xplat(will run on mono for now) and their are also efforts underway from some time now to run roslyn xplat on coreclr/mono too.
So it looks like currently this issue is swinging in the direction of msbuild+roslyn+mono.

@Alexx999

This comment has been minimized.

Contributor

Alexx999 commented Feb 9, 2015

hmmm, I believe there is "Compilation" test for CoreClr which actually compiles some code into assembly. Not sure if it's working on Unix though.
So I believe it will be best to use some kind of self-hosted process - you can build mscorlib using Roslyn running on CoreClr 😄
And to break dependency loop - isn't is possible to have pre-built mscorlib somewhere? As I understand it's purely managed, so it should be binary compatible with source-built runtime.
So the workflow will look like: checkout clean source - use pre-built mscorlib to create fully working CoreClr - make some modifications and use previously built CoreClr to make a build
And if I remember it right - mono has some difficulties during build too (with similar reasons).

@shahid-pk

This comment has been minimized.

Contributor

shahid-pk commented Feb 9, 2015

@Alexx999
I think while building mscorlib you simply do not reference any assembly. so their is no dependency because i do not see mscorlib referencing anything from other managed assemblies i may be wrong. The problem mono guys were facing was some assemblies in the BCL that were cross referencing each other (i can not recall those assemblies).It was problem during build because one of them needed to be build before building the other. They solved that problem in three steps first their build compiles one of the assembly without the parts that reference the other assembly then they compile the second assembly which references the first assembly completely and thirdly they recompile the first assembly with the parts referencing the second assembly.

@Alexx999

This comment has been minimized.

Contributor

Alexx999 commented Feb 9, 2015

@shahid-pk actually no, they too have C# compiler written in C#, so you can't compile Mono without having Mono

@shahid-pk

This comment has been minimized.

Contributor

shahid-pk commented Feb 9, 2015

@Alexx999
yes their c# compiler is written in c# and they require c# to compile their compiler not mono.
again because mono's csharp compiler or mcs does not require any managed assembly it only requires a c# compiler.
To solve this problem they use monolite a precompiled unmanaged c# compiler(which gets downloaded when you build mono locally and you do not have mono/mcs) for compiling mcs during the build and when mcs is compiled they use the same to compile all the managed assemblies.

@akoeplinger

This comment has been minimized.

Member

akoeplinger commented Feb 10, 2015

@shahid-pk

To solve this problem they use monolite a precompiled unmanaged c# compiler ..

monolite isn't unmanaged at all, it's just a small managed version of mcs + some class libs that contains only the stuff necessary to compile mcs on the newly-built Mono runtime. The cyclic dependencies (System.dll -> System.Xml.dll -> System.dll) are indeed an issue that is solved by doing multiple build stages as you said.

OnTopic: I just got the mscorlib project building on Mono with xbuild by commenting out a few things, I'll take a look tomorrow what can be upstreamed. I agree that xbuild has some rough edges so it may still be better to move to another solution that only invokes mcs.

@shahid-pk

This comment has been minimized.

Contributor

shahid-pk commented Feb 10, 2015

@akoeplinger
thanks for the correction about monolite.And great to hear that you got xbuild to compile mscorlib. waiting for your pr.

@kangaroo

This comment has been minimized.

Contributor

kangaroo commented Feb 10, 2015

Relying on mono is not a good strategy. We should bootstrap off a binary available BCL including the contract base to have Roslyn work, and self-host.

@ellismg

This comment has been minimized.

Contributor

ellismg commented Feb 10, 2015

@akoeplinger I would be interested in understanding what you do for the BclRewriter step. Do you run the tool on Mono, if so, does the PDB generation work (I would assume not as IIRC, it will use the COM APIs to write PDBs).

Not running that tool will be problematic long term as it could cause us to not remove FCall and QCalls that the runtime will expect to be missing and I'm not sure we want to spend the effort to get the runtime to support something like that. We could consider something like not generating a PDB for now, but we need to come up with a long term strategy since I think we want to use this tool long term.

@OtherCrashOverride

This comment has been minimized.

OtherCrashOverride commented Feb 10, 2015

I too am interested to know the long term strategy for bootstrapping the runtime on platforms where a managed runtime does not already exist. In addition to depending on mono or requiring a set of pre-compiled binaries (like monolite), there is also another option: a native compiler such as csc in Rotor. The idea being that its 'just-enough-compiler' to build the runtime to a state where a production compiler could take over.

Since the pre-compiled binaries approach sounds like the most viable short term method, I would like to take this opportunity to point out that its important that I am able to build any arbitrary revision of the code forever. This is where monolite falls down and you have to hack build scripts to find older binary versions.

@kangaroo

This comment has been minimized.

Contributor

kangaroo commented Feb 10, 2015

@OtherCrashOverride To be honest, supporting a native compiler than can build any current and future codebase is nuts. Its much better to bootstrap from a managed base. Someone just needs to step up and not have it be 'get-monolite-latest', and have it be 'get-bootstrap-version-i-need'. In this day and age, given BCL size, we're talking megabytes of S3 storage+transfer over years. MS could even CDN cache it.

Lets not make work for the sake of making work. Someone just needs to step up and pay the hosting costs, hopefully MS will do so.

@kangaroo

This comment has been minimized.

Contributor

kangaroo commented Feb 10, 2015

@OtherCrashOverride To be clear, the 'hack build scripts' problem is solved by someone stepping up to pay for hosting. If microsoft steps up, I'll commit to personally mirroring in the case that they go out of business.

@OtherCrashOverride

This comment has been minimized.

OtherCrashOverride commented Feb 10, 2015

To be honest, supporting a native compiler than can build any current and future codebase is nuts.

We call that a 'least desirable solution' and I completely agree that it is. Its only presented so that future generations reading this will know that it was considered and why it (most likely) was not chosen.

We should also consider the approach that C/C++ uses. Since LLVM is ubiquitous, we could simply support 'cross compiling' the entirety of the runtime native and managed code.

@ellismg

This comment has been minimized.

Contributor

ellismg commented Feb 10, 2015

@akoeplinger BTW, I re-read my message just now and it seemed like it came across as a bit of a downer. I'm really excited that you're putting in the effort to getting mscorlib.dll building on Linux with mcs and xbuild and I'm excited to see what can be pulled upstream. Do let me know if you need any help or if you have a set of changes you want me to look at. I think in the short term we're willing to do some hacks here if it gets us a better experience than "copy a bunch of files from Windows"..

@kangaroo

This comment has been minimized.

Contributor

kangaroo commented Feb 10, 2015

@ellismg Relying on mono long-term is going to cause multitudes of pain. We should solve the 'monolite' problem they solved sooner than later. It all boils down to emitting enough of the bcl to let roslyn run and bootstrap the rest.

We've already started deps discussions on gettext, now on mono, where does it end?

There is no benefit using mcs over a bootstrapped roslyn. Can MS really not provide enough azure hosting to host the binaries as breaking changes happen?

@ellismg

This comment has been minimized.

Contributor

ellismg commented Feb 10, 2015

@kangaroo I'm sorry if it sounded like I was proposing that we run on Mono long term. For what you're proposing, are you thinking Roslyn on Mono or Roslyn on CoreCLR? Long term, we'd like to get to Roslyn on CoreCLR (and at that point I think it would be very reasonable for us to provide binaries for bootstrapping), but I think there's a bunch of work to between here and there and if we're able to remove the windows requirement for building mscorlib sooner by adopting mono/mcs temporarily, I just wanted to make it clear that we would be open to that.

@kangaroo

This comment has been minimized.

Contributor

kangaroo commented Feb 10, 2015

@ellismg Roslyn on CoreCLR. Relying on an external project for longterm viability just won't work. Sure there is a bunch of work, but its not a mountain. Likely a pile. The sooner decision to tackle the pile is build strategy. msbuild has advantages due to it already having projects, but triple-depending the bootsrap on 'clr that can build corlib to run roslyn AND some msbuild variant' seems nuts.

IMO -- We should bite the bullet and move the mscorlib build to cmake and bootstrap off that.

Thoughts?

@ellismg

This comment has been minimized.

Contributor

ellismg commented Feb 10, 2015

@kangaroo Okay. Roslyn on CoreCLR is the the long term direction we are moving towards. On the framework side, @stephentoub and I are working on bringing up all the necessary libraries that Roslyn needs. We are doing as much of that work in the open (on CoreFx and here) as possible. For some dependencies we haven't yet open sourced the code and we are working hard to get it out there so we can continue to work on GitHub. Steve and I are working with Roslyn folks to get someone from their team to help with the Roslyn side of things. That should spin up this week (hopefully) and of course all of that work will happen in the open on dotnet/roslyn.

You're right about the build system, that's a bigger issue and we're still trying to figure out internally what the right long term direction is. I would hate for us to pivot to CMake just to move again in a few months, so I'm biased towards sticking with MSBuild + XBuild for now. Especially when you consider that whatever you do for mscorlib you'll need to do for most if not all of CoreFx as well, since long term most of the stack is going to be produced out of there.

I'd be interested in what @AlexGhiondea and @weshaggard think about this.

@kangaroo

This comment has been minimized.

Contributor

kangaroo commented Feb 10, 2015

@ellismg If we want to rely on msbuild I see two options:

1: Open source msbuild
2: Fork xbuild and maintain/improve it internally as a hosted build tool

Both of these suck and seem like a ton of work, that said I agree. Lets not jump from the frying pan into the fire, but lets not rely on xbuild.

@ellismg

This comment has been minimized.

Contributor

ellismg commented Feb 10, 2015

@kangaroo Completely agree on all points. So you understand my aim, I'm trying to get some story for folks that don't have access to Windows and want to play around with the managed portions of the runtime that isn't fetch mscorlib from @ellismg's docker image or @kangaroo's dropbox.

I'm looking for the path of least resistance to get to that state today that minimizes throw away work. To the extent where we can leverage what the mono folks have while we bring up our stack and figured out our long term plans, I want to do. But I'm not wed to any particular technology stack long term, I just want to do what's right for the project at each point in time.

It sounds like you feel the same way, which I appreciate.

Thank you so much for all of your hard work here, by the way. It's really great to see CoreCLR booting up on OS X again.

@OtherCrashOverride

This comment has been minimized.

OtherCrashOverride commented Feb 10, 2015

I would also like to request that consideration be given that what ever solution is implemented that it support offline bootstrapping. I should be able to bring up any arbitrary platform without having to also monitor who its talking to and whether they can be trusted. I need a complete network-less method where all component and checksums are known in advance and will never change due to 3rd parties.

@Alexx999

This comment has been minimized.

Contributor

Alexx999 commented Feb 10, 2015

Well, offline bootstrap can work like:

  1. You have any platform that has working CoreClr
  2. You set the defines that you need and build mscorlib for new platform
    I'm not sure if it's really possible to make it fully offline without second system. Imagine yourself on system without C/C++ compiler - what you gonna do?
@OtherCrashOverride

This comment has been minimized.

OtherCrashOverride commented Feb 10, 2015

The dream scenario is "git clone coreclr" brings in everything required to build it for the platforms it supports, and I can freeze that in time and build it forever. If I am Sony, for example, and hackers have breached the network so the entirety of internet access has been been cut corporate wide, I should still be able to build coreclr. (The reality has more to do with the lessons learned from monolite than hypothetical nightmare scenarios.)

I would suggest that mscorlib.dll be made universal across platforms since its a managed assembly, Then any build of it would automatically be a 'cross compile'. This would mean that with a working coreclr.dll/.so, you could download a single universal binary package that contains roslyn and enough runtime to compile the current mscorlib.dll.

@OtherCrashOverride

This comment has been minimized.

OtherCrashOverride commented Feb 10, 2015

As for the msbuild/xbuild/cmake debate. Maybe we don't need any of them? The goal is building the mscorlib/corefx libraries which is a subset of features required from a full universal build tool. It may be easier to have a custom data driven tool written in C# (since Rosyln is a already dependency, a working clr is too). Nobody likes the 'yet-another-tool' answer, but a task specific tool would be favorable to bringing in and maintaining mono's xbuild.

@akoeplinger

This comment has been minimized.

Member

akoeplinger commented Feb 10, 2015

@ellismg

I would be interested in understanding what you do for the BclRewriter step. Do you run the tool on Mono, if so, does the PDB generation work (I would assume not as IIRC, it will use the COM APIs to write PDBs).

Yep, I skip it currently as it seemed to "work" without it and the BclRewriter isn't open source (at least I couldn't find it?) so I can't hack on it.

Here's my branch with some workarounds to get it to compile with Mono's xbuild: https://github.com/akoeplinger/coreclr/commits/mono-xbuild

Unfortunately it relies on some recent bug fixes to work at all so the Mono 3.12 release doesn't cut it. You'll need to add the Mono CI packages repo and sudo apt-get install mono-snapshot-latest as well as nuget if you haven't already.
Then run . mono-snapshot mono (the dot is important) to switch the current terminal session to the snapshot package and run ./build.sh unixmscorlib.

I haven't really done any verification other than using the built mscorlib with the HelloWorld demo app.


I agree with some of @kangaroo's points, using MSBuild as the build system isn't ideal when no high-quality OSS implementation exists. Improving xbuild is one option, not sure if Xamarin would like to sponsor some work on getting the kinks ironed out. Moving away from MSBuild to something else is certainly easier here in coreclr since the mscorlib build doesn't really use it extensively, but I guess it's a much larger issue in corefx as from what I've seen they're pretty tied to it.

@AlexGhiondea

This comment has been minimized.

Member

AlexGhiondea commented Feb 11, 2015

@akoeplinger We have to run BCLRewriter because if we don't there is going to be a mismatch between the methods (FCALLs, QCALLs) present in mscorlib and the ones that are present in coreclr.dll.

It will work for most cases but once you call a managed method which FCALLs/QCALLs into a runtime method which is not available in CoreCLR you are going to get a crash.

We are considering a couple of options around this but we don't have a clear answer yet.

@OtherCrashOverride

This comment has been minimized.

OtherCrashOverride commented Feb 11, 2015

We have to run BCLRewriter because if we don't there is going to be a mismatch between the methods (FCALLs, QCALLs) present in mscorlib and the ones that are present in coreclr.dll.

Currently "DllImport" and "Internal" calls are all over the place in the codebase. Maybe they should all be gathered together and refactored into a formal platform interface. Then a static constructor could instantiate the appropriate interface to use at runtime. This would make mscorlib.dll universal across all platforms and also eliminate the 'fixup'. I currently use this method with mono and various arm devices. I find it much preferable to #ifdef littering the code. It allows me to ship a single binary .dll instead of recompiling specifically for each target. A "DllImport" that is never used in code never loads the .dll/.so it references so is benign.

@dhusemann

This comment has been minimized.

dhusemann commented Mar 8, 2015

Would Cake help in this?? https://github.com/cake-build/cake

@ellismg

This comment has been minimized.

Contributor

ellismg commented Mar 19, 2015

A bit of an update. Today we open sourced MSBuild, so I think the direction we'll want to take this is to continue to us MSBuild for the managed components in the tree and work towards getting MSBuild cross platform. I am guessing at first we'll want to run it on top of Mono but as CoreCLR comes online, we would definitely want to be able to run it on top of CoreCLR.

The other major issue would be getting the BCL Rewriter tool open sourced and working cross platform. Today, I think the major issue would be that it wants to use some COM components in Windows in order to write PDB files. In the short term, I think we can disable this code. Long term, we'll probably want to move to using the new debugging format that Roslyn is adding support for which is open source and will be cross platform.

@david-mitchell

This comment has been minimized.

Contributor

david-mitchell commented Mar 31, 2015

Now that the "xplat" branch of has been published, I've been trying to get msbuild to run on mono in order to build mscorlib on Linux.

The first issue that I encountered is that the project file attempts to download NuGet but fails to run it because the DownloadFile task doesn't know to change the Windows-style path it is given to a Unix-style path.

There is a proposal to add a property function to MSBuild that would perform this transformation, but I don't know if it will actually be adopted or how long it would be before the method would be available for use in the coreclr build process. Because of this, it might be better to copy the necessary path fixing code from MSBuild and dump it into the DownloadFile task.

Thoughts?

A related question: should all discussion of getting mscorlib to build on Linux/OS X occur in this issue, or should new issues be created for "sub-issues" like this one?

@JeremyKuhne

This comment has been minimized.

Member

JeremyKuhne commented Mar 31, 2015

I wonder if we could just start using forward slash in our project files? Afaik the only place it matters in Windows is for UNC paths (e.g. \server\share) and drive rooted paths (e.g. C:\Foo). Neither of which should be used in a project file.

@david-mitchell

This comment has been minimized.

Contributor

david-mitchell commented Mar 31, 2015

@JeremyKuhne that approach could work with handwritten msbuild projects (like dir.targets), but I imagine it would be a terrible experience if we needed to extend it to anything in msbuild.csproj: anyone using Visual Studio to edit the project would have to constantly fight their tooling.

@JeremyKuhne

This comment has been minimized.

Member

JeremyKuhne commented Mar 31, 2015

@david-mitchell wouldn't you always have to deal with VS paths being incorrect?

@david-mitchell

This comment has been minimized.

Contributor

david-mitchell commented Mar 31, 2015

Yes. My point (which was poorly articulated) is that changing the path style works well for portions of the project that are handwritten and only depend on other handwritten portions of the project.

However, if there are handwritten portions of the project that depend on VS-generated project files, then the approach would not work well.

At this point, I should probably just change the paths necessary for the DownloadFile task and see how much farther I get; I could be worrying over nothing.

@JeremyKuhne

This comment has been minimized.

Member

JeremyKuhne commented Mar 31, 2015

Ah, I see. Could you use Path.GetFullPath directly for such scenarios? Or do we need a function that doesn't resolve the path (e.g. works with relative paths)?

FYI: As a quick sanity check I flipped all of the build files in CoreFx to forward slash and built fine in Windows.

@david-mitchell

This comment has been minimized.

Contributor

david-mitchell commented Mar 31, 2015

Path.GetFullPath will not convert \ to / on Unix systems because \ is a valid (although arguably ridiculous) file name character.

@JeremyKuhne

This comment has been minimized.

Member

JeremyKuhne commented Mar 31, 2015

Sorry, new to Unix. :) \ being valid makes generalization really hard. :/

How about just doing a simple char replace? $(Foo.Replace('\', '/'))

I'm not necessarily opposed to getting a method added. The trick here is not to take a dependency on the open source MSBuild, and as such we should probably add said method to the .NET build tools dll. (I'm still for adding/exposing this method to MSBuild to solve for the long term.)

That doesn't, however, handle bootstrapping the build tools. Forward slashing and or .Replace in the build environment should be enough to start I think.

@david-mitchell

This comment has been minimized.

Contributor

david-mitchell commented Mar 31, 2015

@JeremyKuhne Replace sounds like a "good enough for now" plan. I should have some time to play with it and submit a PR this evening (Pacific Time) if no one else gets to it first.

@akoeplinger

This comment has been minimized.

Member

akoeplinger commented Mar 31, 2015

If you're just concerned about bootstrapping/downloading NuGet, it may be easier to just shell out to curl like msbuild does in the xplat branch: https://github.com/Microsoft/msbuild/blob/6a8d6e20bcfee1e0f1114dfd3a8459d1c3ff87a3/dir.targets#L54

@JeremyKuhne

This comment has been minimized.

Member

JeremyKuhne commented Mar 31, 2015

@david-mitchell here is one approach:

  <PropertyGroup>
    <TestPath>../../Foo.dll</TestPath>
    <DirectorySeparator Condition="'$(DirectorySeparator)'==''">$([System.IO.Path]::DirectorySeparatorChar)</DirectorySeparator>
    <AltDirectorySeparator Condition="'$(AltDirectorySeparator)'==''">$([System.IO.Path]::AltDirectorySeparatorChar)</AltDirectorySeparator>
  </PropertyGroup>
  <Target Name="Test">
    <Message
       Importance="high"
       Text="Path: $(TestPath.Replace($(AltDirectorySeparator),$(DirectorySeparator)))"
       />
  </Target>

I'm, of course, assuming this would work on Unix if you flip the slashes.

@david-mitchell

This comment has been minimized.

Contributor

david-mitchell commented Apr 1, 2015

@JeremyKuhne AltDirectorySeparator returns / on mono under Linux, so it cannot be used in this instance. However, I have found a similar, workable solution (see the recent pull request).

@ghost

This comment has been minimized.

ghost commented Jun 2, 2015

@ellismg, @AlexGhiondea, is there any news around making BCLRewriter (or something similar) open source?

In case I am not mistaken, this is the last piece missing to make build system xplat. It is a hassle for people on other platforms to build and test latest greatest CoreCLR bits (case in hand #1044 (comment))

@ellismg

This comment has been minimized.

Contributor

ellismg commented Aug 20, 2015

Fixed with 4df67c7.

BclRewriter is not yet open source but we are working on getting it ready to be open sourced into build tools. For now we restore a closed source copy that can run on Mono.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment