Add a Cargo-based build system to eventually replace make

[alexcrichton]

This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:

1. The ./configure script is run with --enable-rustbuild and other standard
   configuration options.
2. A Makefile is generate which proxies commands to the new build system.
3. The new build system has a Python script entry point which manages
   downloading both a Rust and Cargo nightly. This initial script also manages
   building the build system itself (which is written in Rust).
4. The build system, written in rust and called bootstrap, architects how to
   call cargo and manages building all native libraries and such.

One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:

• The makefiles are impenetrable to all but a few people on this
  planet. This means that contributions to the build system are almost
  nonexistent, and furthermore if a build system change is needed it's
  incredibly difficult to figure out how to do so. This hindrance prevents us
  from doing some "perhaps fancier" things we may wish to do in make.
• Our build system, while portable, is unfortunately not infinitely portable
  everywhere. For example the recently-introduced MSVC target is quite unlikely
  to have make installed by default (e.g. it requires building inside of an
  MSYS2 shell currently). Conversely, the portability of make comes at a cost of
  crazy and weird hacks to work around all sorts of versions of software
  everywhere, especially when it comes to the configure script and makefiles.
  By rewriting this logic in one of the most robust platforms there is, Rust,
  we get to assuage all of these worries for free!
• There's a standard tool to build Rust crates, Cargo, but the standard library
  and compiler don't use it. This means that they cannot benefit easily from the
  crates.io ecosystem, nor can the ecosystem benefit from a standard way to
  build this repository itself. Moving to Cargo should help assuage both of
  these needs. This has the added benefit of making the compiler more
  approachable for newbies as working on the compiler will just happen to be
  working on a large Cargo project, all the same standard tools and tricks will
  apply.
• There's a huge amount of portability information in the main distribution, for
  example around cross compiling, compiling on new OSes, etc. Pushing this logic
  into standard crates (like gcc) enables the community to immediately benefit
  from new build logic.

Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:

1. Land a second build system in-tree that can be itereated on an and
   contributed to. This will not be used just yet in terms of gating new commits
   to the repo.
2. Over time, bring the second build system to feature parity with the old build
   system, start setting up CI for both build systems.
3. At some point in the future, switch the default to the new build system, but
   keep the old one around.
4. At some further point in the future, delete the entire old build system.

---

Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the src/bootstrap
directory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:

• Compiling LLVM with cmake instead of ./configure + make. The LLVM
  project is removing their autotools build system, so we'd have to make this
  transition eventually anyway.
• Compiling compiler-rt with cmake as well (for the same rationale as above).
• Adding Cargo.toml to map out the dependency graph to all crates, and also
  adding build.rs files where appropriate. For example alloc_jemalloc has a
  script to build jemalloc, flate has a script to build miniz.c, std will
  build libbacktrace, etc.
• Orchestrating all the calls to cargo to build the standard distribution,
  following the normal bootstrapping process. This also tracks dependencies
  between steps to ensure cross-compilation targets happen as well.
• Configuration is intended to eventually be done through a config.toml file,
  so support is implemented for this. The most likely vector of configuration
  for now, however, is likely through config.mk (what ./configure emits), so
  the build system currently parses this information.

There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86_64 and x86, but more portability is always
welcome!

---

Future functionality left to implement

• Re-verify that multi-host builds work
• Verify android build works
• Verify iOS build work (mostly compiler-rt)
• Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc
• Implement testing -- this is a huge bullet point with lots of sub-bullets
• Build and generate documentation (plus the various tools we have in-tree)
• Move various src/etc scripts into Rust -- not sure how this interacts with make build system
• Implement make install - like testing this is also quite massive
• Deduplicate version information with makefiles

[rust-highfive]

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

[sfackler]

🎊

[steveklabnik]

Hero mode omg 💯 💯 💯

[alexcrichton]

In terms of review, I've tried to split things up in as bite-sized chunks as possible, and I'll avoid rebasing for awhile so the commit comments are preserved

[jonas-schievink]

Yesss ✌️

[nikomatsakis]

Hold the phone.

[aturon]

@alexcrichton does it again!

[retep998]

Is the python bit optional? As in, if we already had a recent enough rust/cargo nightly, could we just directly build the Rust bootstrap and go from there?

[frewsxcv]

if not quiet:

[alexcrichton]

Oh this is actually intended (albeit kinda weirdly) where quiet means "just print what you're downloading" and not quiet means "print the command we're running"

[frewsxcv]

if not quiet

[alexcrichton]

ah same as above where quiet means "less status messages" and not quiet means "lots of status messages"

I'll update this variable name though

[frewsxcv]

if "/" not in p:

[frewsxcv]

Not a huge deal, but FYI, filter here is shadowing the global filter function

[frewsxcv]

This else is not necessary

[frewsxcv]

I don't think iter(..) is necessary here (nor a few lines down), but I could be wrong

[frewsxcv]

assert is a statement, not a function (so you can drop the parens)

[frewsxcv]

Might be good to use a context manager here (with open(..):) so the file gets properly closed

[frewsxcv]

Use sys.exit instead of exit. The latter is something unrelated and misleading

https://docs.python.org/2/library/sys.html#sys.exit

https://docs.python.org/2/library/constants.html#exit

[frewsxcv]

This could also be written:

if cputype in {'i386', 'i486', 'i686', 'i786', 'x86'}:

{a, b, c} is set notation fyi

[frewsxcv]

Ditto earlier comment about using a context manager with open

[frewsxcv]

Ditto earlier comment about using a context manager with open

[frewsxcv]

Missing an endif in this conditional now

[alexcrichton]

Oops good catch.

[alexcrichton]

@retep998

Is the python bit optional? As in, if we already had a recent enough rust/cargo nightly, could we just directly build the Rust bootstrap and go from there?

There's some discussion here about this as well, but I wouldn't necessarily want it to be a prerequisite to build Rust that you have Rust installed just yet. Maybe down the road we can expect that (I think gcc requires this, right?).

That being said I would love to remove even the python script (as @frewsxcv has discovered I'm awful at writing python) and just have pure Rust from the get-go. For now though we require Python because of the LLVM build anyway, and I tried to keep the Python as tiny as possible. Hopefully we can either minimize the python over time (e.g. get it even smaller than it is today) or possibly even rewrite it in Rust as you mentioned and just assume that rustc is available.

[alexcrichton]

I've started keeping a tentative list of items left-to-implement at the top of this PR as well. I plan on explicitly not implementing them here but rather opening follow-up tracking issues for them.

[lndbrg]

RuntimeError

[lndbrg]

ValueError

[lndbrg]

ValueError

[rkruppe]

Aside: How does one use this build system? I tried ./configure --enable-rustbuild (64 bit Windows in a mingw shell) but that fails in a clean checkout:

configure: looking at LLVM
configure:
configure: configuring LLVM for x86_64-w64-mingw32
configure: configuring LLVM with:
configure: --enable-targets=x86,x86_64,arm,aarch64,mips,powerpc --enable-optimized --disable-assertions --disable-docs --enable-bindings=none --disable-terminfo --disable-zlib --disable-libffi --disable-pthreads --build=x86_64-w64-mingw32                         --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-python=/mingw64/bin/python2.7
./configure: line 1738: <...>/src/llvm/configure: No such file or directory

src/llvm is empty, not even a .git. It seems that --enable-rustbuild disables checking out submodules.

[alexcrichton]

I'm gonna rebase this to get rid of the merge and [rebase] commits, and pray nothing dies anywhere!

[alexcrichton]

@bors: r=brson 1bd4a65

I guess that wasn't so bad after all

[alexcrichton]

@bors: r=brson 55dd595

[bors]

⌛️ Testing commit 55dd595 with merge 2300556...

[bors]

💔 Test failed - auto-mac-64-opt

[alexcrichton]

@bors: retry

On Thu, Feb 11, 2016 at 2:16 PM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-mac-64-opt
http://buildbot.rust-lang.org/builders/auto-mac-64-opt/builds/8008

—
Reply to this email directly or view it on GitHub
#31123 (comment).

[bors]

⌛️ Testing commit 55dd595 with merge 78a5d5b...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt

[aturon]

🍻