Moving Rover (dotnet-bootstrapping tool) from private repository into dotnet/core

[ghost]

Tool itself is functional, but need to build the testing pieces.

Going to start a list here, we've got good discussions and feedback happening, so I thought I'd just keep the work centralized here so that everyone is in the loop as to what I am up to (I click the checkbox as I accomplish the work - pushes will be intermittent throughout the day; order is determined by mood):

• Explicitly copy just the binaries listed in the target repository directory (at the moment I am using the Kleene Star to select all the so files to copy) [ I'd like to add in a 'patching' feature that will just adopt the layout of the target directory and then move files from a source path; not this week though]
• Incorporate Run script for the docker containers (i.e. something that will just launch them and execute my test case)
• Use DotNet Bot + QOTD as test case
• Incorporate intended Docker scripts
• Provide examples in the README.md
• Document the layout of the working directory
• Do a 'satisfaction pass' over the documentation to make it... cleaner. [ used format of man pages ]
• Place .gitattributes/.gitignore in root (if not there); crib (great word) from CoreFX
• Remove dependency installer
• (Regarding colored text) Guard writing the escape codes with a check for sys.stdout.isatty()
• (Minor bug fix) - On Unexpected Rover Exception I am still printing a success message (oops) [ added an os._exit(...) call to the UnexpectedRoverException() method so that after we catch the exception we just bail out. This should prevent any messages from popping up or risk of some future addition executing that maybe depends on this existing ]
• (Minor bug fix) - If interrupted during the cloning of repositories, restarting causes us to skip over the clone step [ I removed the -clone switch, I think this was over-engineered so I just check if the directory exists, if it doesn't then I will clone ]
• Add a -to that overrides the default directory name that we stage in. Currently we stage in a local directory with the format 'dotnet-'
• Cache CLI download
• sudo repro-script on creation (so long as uid == 0)

[ellismg]

It may make sense to move .gitignore and .gitattributes to the top level of the repository (and maybe just crib from the version we have in corefx)?

[ghost]

test ci please

[ghost]

CI test is passing, going to hold off on one more thing - I only want the CI to run when modifying dotnet.bootstrap.py

[tmds]

I tried the dotnet.bootstrap.py tool on Fedora 25.

What I found out:

• The tool does not take care of dependencies, which is okay because each repo does a pretty good job of reporting it's own dependencies. It means "development mode" is not exceptional but it is a typical part of bootstrapping. Why isn't this the 'only' mode?
• The script files which are created to debug a failing build step miss the execute flag. They need a manual chmod +x.
• Each time I restarted the tool, it downloaded the cli again saying "Could not locate a payload at path ''". Perhaps the tool can store the downloaded file as dotnet.tar.gz file and automatically use it if no other payload was provided?
• I don't get what the -nopatch is for. The goal of the tool is to patch? Or perhaps I don't understand what the nopatch does.
• I had to make some changes to the repositories which were already fixed on master due to the compiler being a more recent one. This is a normal part of bootstrapping.
• In the end, I had a dotnet. It managed to 'dotnet new' and 'dotnet restore'. But the 'dotnet build' hang. It was using msbuild. This was a bit of a surprise since there hasn't been an SDK release with msbuild. How does the tool chose a tar.gz file to download? And how does it chose the tags/branches of the different repositories it is using?
• At that point I decided to manually pick the fedora LTS release and provide it using the -payload option. It took me a bit longer to find on the dot.net website that it would for a Windows/Mac user. When running it providing the payload option I got an uncaught exception: "SemanticVersion instance has no attribute 'find'" of type <type 'exception.AttributeError'>.
• Not sure what to delete, I deleted the fedora.25-x64-dotnet folder to get things in order again. I didn't expect the src folder to be in there, so this accidentally caused me to delete the changes I made to the repositories. Perhaps the src folder can move up one level?

All in all, I liked how the tool helps bootstrapping. Using the scripts, it has a nice workflow a la "bootstrap --continue".

[ghost]

@tmds Fantastic feedback! Thank you!

• "The tool does not take care of dependencies" - True, this was snipped off (too many instances out there where dependencies can't just be 'guaranteed' which makes the feature useless except for cases that are 'well known', but in the well known instances we are less likely to need this. If really in a bind for dependencies, review the docker containers in base/lab/containers that has the apt-get, yum, zypper, dnf, etc. commands to pull in packages. @gkhanna79 (Relevant to you).

• "The script files which are created to debug a failing build step miss the execute flag" - Will fix.

• "It downloaded the CLI again..." - Will fix.

• "What is no patch?" - This is an experimental feature that probably does not need to exist any longer (i.e. I'll remove it after a bit more review).

• "How does the tool choose?" - At the moment it does no choosing - we simply grab the latest debian dotnet bits, then we grab specific checkout commits and build, patch, run. In the future we will take the version number/commit hash from the binary and build only those pieces (in other words, whatever binary versions you provide (via -payload; or whatever we download) we will reproduce for the selected platform).

• Given your story the DevMode makes me think this has caused a headache having this distinction. I too think DevMode should be 'default' now - or at least removed. I also like your idea of 'bootstrap --continue' - probably going to steal that ;)

Again, thank you for sharing your experience with using this tool!

[tmds]

Second attempt of bootstrapping on Fedora 25.

Install Fedora 25

Used iso: https://download.fedoraproject.org/pub/fedora/linux/releases/25/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-25-1.3.iso

Install dependencies

$ sudo dnf -y install python cmake clang llvm libicu-devel lldb-devel libunwind-devel lttng-ust-devel libuuid-devel zlib-devel libcurl-devel krb5-devel openssl-devel automake libtool

Get Rover

$ git clone https://github.com/dotnet/core
$ cd core
$ git checkout bootstrap
$ git rev-parse HEAD
f895ac6b595ff03a22bd1278f803dbfe29c61e75
$ cd tools/dotnet-bootstrap

Get payload SDK

Using Fedora 24 .NET Core 1.1 SDK:

$ curl -sSL -o dotnet.tar.gz https://go.microsoft.com/fwlink/?LinkID=835025

Select your branches

$ mkdir -p fedora.25-x64-dotnet/src
$ cd fedora.25-x64-dotnet/src
$ git clone https://github.com/dotnet/corefx
$ cd corefx
$ git checkout release/1.1.0
$ git rev-parse HEAD
639eeadbf43c0b2be334ea0ee83ae9da2ca03578
$ cd ..
$ git clone https://github.com/dotnet/coreclr
$ cd coreclr
$ git checkout release/1.1.0
$ git rev-parse HEAD
673561935b3112e1128392b61b4e974f1b04545c
$ cd ../..

I let rover pick branches for libuv and core-setup:

core-setup at v1.1.0 (928f77c4bc3f49d892459992fb6e1d5542cb5e86)
libuv at v1.9.0 (229b3a4cc150aebd6561e6bd43076eafa7a03756)

Bootstrap

$ ./dotnet.bootstrap.py -payload dotnet.tar.gz
...
** ROVER spawned a 'dotnet' in ./fedora.25-x64-dotnet/bin/(enjoy!)

Does it work?

Edited:
First time I tried this dotnet new;dotnet restore;dotnet run got stuck compiling.
Retried everything from scratch after merge (core at 1f2bab7) with same versions for coreclr, corefx, libuv and core-setup and it was working fine. Unclear what the root cause was for the fail.

[ellismg]

This PR also adds a ubuntu directory with submodules. Was that expected or was output from a local invocation of the tool accidentally included?

[ghost]

@ellismg Accident. I will fix this up and close it up now actually.

[ghost]

@tmds Oops, I do actually have the check for chmod +x on the repro scripts. If you run the script in sudo mode this is the only time I will set the X bit on the repro script.

[ghost]

test ci please