Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Cargo-based build system to eventually replace make #31123

Merged
merged 10 commits into from Feb 12, 2016

Conversation

@alexcrichton
Copy link
Member

commented Jan 22, 2016

This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:

  1. The ./configure script is run with --enable-rustbuild and other standard
    configuration options.
  2. A Makefile is generate which proxies commands to the new build system.
  3. The new build system has a Python script entry point which manages
    downloading both a Rust and Cargo nightly. This initial script also manages
    building the build system itself (which is written in Rust).
  4. The build system, written in rust and called bootstrap, architects how to
    call cargo and manages building all native libraries and such.

One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:

  • The makefiles are impenetrable to all but a few people on this
    planet. This means that contributions to the build system are almost
    nonexistent, and furthermore if a build system change is needed it's
    incredibly difficult to figure out how to do so. This hindrance prevents us
    from doing some "perhaps fancier" things we may wish to do in make.
  • Our build system, while portable, is unfortunately not infinitely portable
    everywhere. For example the recently-introduced MSVC target is quite unlikely
    to have make installed by default (e.g. it requires building inside of an
    MSYS2 shell currently). Conversely, the portability of make comes at a cost of
    crazy and weird hacks to work around all sorts of versions of software
    everywhere, especially when it comes to the configure script and makefiles.
    By rewriting this logic in one of the most robust platforms there is, Rust,
    we get to assuage all of these worries for free!
  • There's a standard tool to build Rust crates, Cargo, but the standard library
    and compiler don't use it. This means that they cannot benefit easily from the
    crates.io ecosystem, nor can the ecosystem benefit from a standard way to
    build this repository itself. Moving to Cargo should help assuage both of
    these needs. This has the added benefit of making the compiler more
    approachable for newbies as working on the compiler will just happen to be
    working on a large Cargo project, all the same standard tools and tricks will
    apply.
  • There's a huge amount of portability information in the main distribution, for
    example around cross compiling, compiling on new OSes, etc. Pushing this logic
    into standard crates (like gcc) enables the community to immediately benefit
    from new build logic.

Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:

  1. Land a second build system in-tree that can be itereated on an and
    contributed to. This will not be used just yet in terms of gating new commits
    to the repo.
  2. Over time, bring the second build system to feature parity with the old build
    system, start setting up CI for both build systems.
  3. At some point in the future, switch the default to the new build system, but
    keep the old one around.
  4. At some further point in the future, delete the entire old build system.

Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the src/bootstrap
directory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:

  • Compiling LLVM with cmake instead of ./configure + make. The LLVM
    project is removing their autotools build system, so we'd have to make this
    transition eventually anyway.
  • Compiling compiler-rt with cmake as well (for the same rationale as above).
  • Adding Cargo.toml to map out the dependency graph to all crates, and also
    adding build.rs files where appropriate. For example alloc_jemalloc has a
    script to build jemalloc, flate has a script to build miniz.c, std will
    build libbacktrace, etc.
  • Orchestrating all the calls to cargo to build the standard distribution,
    following the normal bootstrapping process. This also tracks dependencies
    between steps to ensure cross-compilation targets happen as well.
  • Configuration is intended to eventually be done through a config.toml file,
    so support is implemented for this. The most likely vector of configuration
    for now, however, is likely through config.mk (what ./configure emits), so
    the build system currently parses this information.

There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86_64 and x86, but more portability is always
welcome!


Future functionality left to implement

  • Re-verify that multi-host builds work
  • Verify android build works
  • Verify iOS build work (mostly compiler-rt)
  • Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc
  • Implement testing -- this is a huge bullet point with lots of sub-bullets
  • Build and generate documentation (plus the various tools we have in-tree)
  • Move various src/etc scripts into Rust -- not sure how this interacts with make build system
  • Implement make install - like testing this is also quite massive
  • Deduplicate version information with makefiles
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Jan 22, 2016

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

@sfackler

This comment has been minimized.

Copy link
Member

commented Jan 22, 2016

🎊

@steveklabnik

This comment has been minimized.

Copy link
Member

commented Jan 22, 2016

Hero mode omg 💯 💯 💯

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Jan 22, 2016

In terms of review, I've tried to split things up in as bite-sized chunks as possible, and I'll avoid rebasing for awhile so the commit comments are preserved

@jonas-schievink

This comment has been minimized.

Copy link
Member

commented Jan 22, 2016

Yesss ✌️

@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Jan 22, 2016

Hold the phone.

@aturon

This comment has been minimized.

Copy link
Member

commented Jan 22, 2016

@alexcrichton does it again!

@retep998

This comment has been minimized.

Copy link
Member

commented Jan 22, 2016

Is the python bit optional? As in, if we already had a recent enough rust/cargo nightly, could we just directly build the Rust bootstrap and go from there?

import tarfile

def get(url, path, quiet=False):
if quiet:

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

if not quiet:

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Jan 23, 2016

Author Member

Oh this is actually intended (albeit kinda weirdly) where quiet means "just print what you're downloading" and not quiet means "print the command we're running"

run(["curl", "-o", path, url], quiet=quiet)

def unpack(tarball, dst, quiet=False, filter=None):
if quiet:

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

if not quiet

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Jan 23, 2016

Author Member

ah same as above where quiet means "less status messages" and not quiet means "lots of status messages"

I'll update this variable name though

fname = os.path.basename(tarball).replace(".tar.gz", "")
with contextlib.closing(tarfile.open(tarball)) as tar:
for p in tar.getnames():
if not "/" in p:

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

if "/" not in p:

else:
run(["curl", "-o", path, url], quiet=quiet)

def unpack(tarball, dst, quiet=False, filter=None):

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

Not a huge deal, but FYI, filter here is shadowing the global filter function

name = p.replace(fname + "/", "", 1)
if filter is not None and not name.startswith(filter):
continue
else:

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

This else is not necessary

return os.path.join(self.build_dir, self.build, "stage0")

def get_toml(self, key):
for line in iter(self.config_toml.splitlines()):

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

I don't think iter(..) is necessary here (nor a few lines down), but I could be wrong

nightlies = os.path.join(self.rust_root, "src/nightlies.txt")
rustc, cargo = open(nightlies, 'r').read().split("\n")[:2]
assert(rustc.startswith("rustc: "))
assert(cargo.startswith("cargo: "))

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

assert is a statement, not a function (so you can drop the parens)


def parse_nightly_dates(self):
nightlies = os.path.join(self.rust_root, "src/nightlies.txt")
rustc, cargo = open(nightlies, 'r').read().split("\n")[:2]

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

Might be good to use a context manager here (with open(..):) so the file gets properly closed

proc = subprocess.Popen(args, env = env)
ret = proc.wait()
if ret != 0:
exit(ret)

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

Use sys.exit instead of exit. The latter is something unrelated and misleading

https://docs.python.org/2/library/sys.html#sys.exit

https://docs.python.org/2/library/constants.html#exit

raise Exception("unknown OS type: " + ostype)

if (cputype == 'i386' or cputype == 'i486' or cputype == 'i686' or
cputype == 'i786' or cputype == 'x86'):

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

This could also be written:

if cputype in {'i386', 'i486', 'i686', 'i786', 'x86'}:

{a, b, c} is set notation fyi

rb.verbose = args.verbose

try:
rb.config_toml = open(args.config or 'config.toml').read()

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

Ditto earlier comment about using a context manager with open

except:
pass
try:
rb.config_mk = open('config.mk').read()

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 22, 2016

Member

Ditto earlier comment about using a context manager with open

else
RUSTFLAGS_rustc_back := --cfg disable_jemalloc
endif

This comment has been minimized.

Copy link
@frewsxcv

frewsxcv Jan 23, 2016

Member

Missing an endif in this conditional now

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Jan 23, 2016

Author Member

Oops good catch.

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Jan 23, 2016

@retep998

Is the python bit optional? As in, if we already had a recent enough rust/cargo nightly, could we just directly build the Rust bootstrap and go from there?

There's some discussion here about this as well, but I wouldn't necessarily want it to be a prerequisite to build Rust that you have Rust installed just yet. Maybe down the road we can expect that (I think gcc requires this, right?).

That being said I would love to remove even the python script (as @frewsxcv has discovered I'm awful at writing python) and just have pure Rust from the get-go. For now though we require Python because of the LLVM build anyway, and I tried to keep the Python as tiny as possible. Hopefully we can either minimize the python over time (e.g. get it even smaller than it is today) or possibly even rewrite it in Rust as you mentioned and just assume that rustc is available.

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Jan 23, 2016

I've started keeping a tentative list of items left-to-implement at the top of this PR as well. I plan on explicitly not implementing them here but rather opening follow-up tracking issues for them.

if code != 0:
if not verbose:
print("failed to run: " + ' '.join(args))
raise Exception("failed to run command")

This comment has been minimized.

Copy link
@lndbrg

lndbrg Jan 23, 2016

RuntimeError

cputype = 'x86_64'
ostype = 'pc-windows-gnu'
else:
raise Exception("unknown OS type: " + ostype)

This comment has been minimized.

Copy link
@lndbrg
elif cputype in {'amd64', 'x86_64', 'x86-64', 'x64'}:
cputype = 'x86_64'
else:
raise Exception("unknown cpu type: " + cputype)

This comment has been minimized.

Copy link
@lndbrg
@rkruppe

This comment has been minimized.

Copy link
Member

commented Jan 23, 2016

Aside: How does one use this build system? I tried ./configure --enable-rustbuild (64 bit Windows in a mingw shell) but that fails in a clean checkout:

configure: looking at LLVM
configure:
configure: configuring LLVM for x86_64-w64-mingw32
configure: configuring LLVM with:
configure: --enable-targets=x86,x86_64,arm,aarch64,mips,powerpc --enable-optimized --disable-assertions --disable-docs --enable-bindings=none --disable-terminfo --disable-zlib --disable-libffi --disable-pthreads --build=x86_64-w64-mingw32                         --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-python=/mingw64/bin/python2.7
./configure: line 1738: <...>/src/llvm/configure: No such file or directory

src/llvm is empty, not even a .git. It seems that --enable-rustbuild disables checking out submodules.

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Feb 11, 2016

I'm gonna rebase this to get rid of the merge and [rebase] commits, and pray nothing dies anywhere!

Add a Cargo-based build system
This commit is the start of a series of commits which start to replace the
makefiles with a Cargo-based build system. The aim is not to remove the
makefiles entirely just yet but rather just replace the portions that invoke the
compiler to do the bootstrap. This commit specifically adds enough support to
perform the bootstrap (and all the cross compilation within) along with
generating documentation.

More commits will follow up in this series to actually wire up the makefiles to
call this build system, so stay tuned!
bootstrap: Read configuration from config.mk
During the transition period where we're still using ./configure and makefiles,
read some extra configuration from `config.mk` if it's present. This means that
the bootstrap build should be configured the same as the original ./configure
invocation.

Eventually this will all be removed in favor of only storing information in
`config.toml` (e.g. the configure script will generate config.toml), but for now
this should suffice.
configure: Remove default NDK path value
This likely isn't always valid, and subverts auto-detection.

@alexcrichton alexcrichton force-pushed the alexcrichton:who-doesnt-want-two-build-systems branch from 96bf1b9 to 1bd4a65 Feb 11, 2016

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Feb 11, 2016

@bors: r=brson 1bd4a65

I guess that wasn't so bad after all

bootstrap: Add a bunch of Cargo.toml files
These describe the structure of all our crate dependencies.
bootstrap: Add build scripts for crates
This commits adds build scripts to the necessary Rust crates for all the native
dependencies. This is currently a duplication of the support found in mk/rt.mk
and is my best effort at representing the logic twice, but there may be some
unfortunate-and-inevitable divergence.

As a summary:

* alloc_jemalloc - build script to compile jemallocal
* flate - build script to compile miniz.c
* rustc_llvm - build script to run llvm-config and learn about how to link it.
  Note that this crucially (and will not ever) compile LLVM as that would take
  far too long.
* rustdoc - build script to compile hoedown
* std - script to determine lots of libraries/linkages as well as compile
  libbacktrace
bootstrap: Add directives to not double-link libs
Have all Cargo-built crates pass `--cfg cargobuild` and then add appropriate
`#[cfg]` definitions to all crates to avoid linking anything if this is passed.
This should help allow libstd to compile with both the makefiles and with Cargo.
rustbook: Make `main` a public function
This will allow it to be used as a crate in a Cargo-based build
rustc_llvm: Tweak how initialization is performed
Refactor a bit to have less repetition and #[cfg] and try to bury it all inside
of a macro.
configure: Add an option to use the cargo build system
This commit adds a `--enable-rustbuild` option to the configure script which
will copy a different `Makefile.in` into place to intercept all `make`
invocations.

Currently this makefile only has one target, but it's expected to be filled out
quite a bit over time!
rustc_back: Fix disabling jemalloc
When building with Cargo we need to detect `feature = "jemalloc"` to enable
jemalloc, so propagate this same change to the build system to pass the right
`--cfg` argument.

@alexcrichton alexcrichton force-pushed the alexcrichton:who-doesnt-want-two-build-systems branch from 1bd4a65 to 55dd595 Feb 11, 2016

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Feb 11, 2016

@bors: r=brson 55dd595

@bors

This comment has been minimized.

Copy link
Contributor

commented Feb 11, 2016

⌛️ Testing commit 55dd595 with merge 2300556...

bors added a commit that referenced this pull request Feb 11, 2016
Auto merge of #31123 - alexcrichton:who-doesnt-want-two-build-systems…
…, r=brson

This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:

1. The `./configure` script is run with `--enable-rustbuild` and other standard
   configuration options.
2. A `Makefile` is generate which proxies commands to the new build system.
3. The new build system has a Python script entry point which manages
   downloading both a Rust and Cargo nightly. This initial script also manages
   building the build system itself (which is written in Rust).
4. The build system, written in rust and called `bootstrap`, architects how to
   call `cargo` and manages building all native libraries and such.

One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:

* The makefiles are impenetrable to all but a few people on this
  planet. This means that contributions to the build system are almost
  nonexistent, and furthermore if a build system change is needed it's
  incredibly difficult to figure out how to do so. This hindrance prevents us
  from doing some "perhaps fancier" things we may wish to do in make.

* Our build system, while portable, is unfortunately not infinitely portable
  everywhere.  For example the recently-introduced MSVC target is quite unlikely
  to have `make` installed by default (e.g. it requires building inside of an
  MSYS2 shell currently). Conversely, the portability of make comes at a cost of
  crazy and weird hacks to work around all sorts of versions of software
  everywhere, especially when it comes to the configure script and makefiles.
  By rewriting this logic in one of the most robust platforms there is, Rust,
  we get to assuage all of these worries for free!

* There's a standard tool to build Rust crates, Cargo, but the standard library
  and compiler don't use it. This means that they cannot benefit easily from the
  crates.io ecosystem, nor can the ecosystem benefit from a standard way to
  build this repository itself. Moving to Cargo should help assuage both of
  these needs. This has the added benefit of making the compiler more
  approachable for newbies as working on the compiler will just happen to be
  working on a large Cargo project, all the same standard tools and tricks will
  apply.

* There's a huge amount of portability information in the main distribution, for
  example around cross compiling, compiling on new OSes, etc. Pushing this logic
  into standard crates (like `gcc`) enables the community to immediately benefit
  from new build logic.

Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:

1. Land a second build system in-tree that can be itereated on an and
   contributed to. This will not be used just yet in terms of gating new commits
   to the repo.
2. Over time, bring the second build system to feature parity with the old build
   system, start setting up CI for both build systems.
3. At some point in the future, switch the default to the new build system, but
   keep the old one around.
4. At some further point in the future, delete the entire old build system.

---

Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the `src/bootstrap`
directory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:

* Compiling LLVM with `cmake` instead of `./configure` + `make`. The LLVM
  project is removing their autotools build system, so we'd have to make this
  transition eventually anyway.

* Compiling compiler-rt with `cmake` as well (for the same rationale as above).

* Adding `Cargo.toml` to map out the dependency graph to all crates, and also
  adding `build.rs` files where appropriate. For example `alloc_jemalloc` has a
  script to build jemalloc, `flate` has a script to build `miniz.c`, `std` will
  build `libbacktrace`, etc.

* Orchestrating all the calls to `cargo` to build the standard distribution,
  following the normal bootstrapping process. This also tracks dependencies
  between steps to ensure cross-compilation targets happen as well.

* Configuration is intended to eventually be done through a `config.toml` file,
  so support is implemented for this. The most likely vector of configuration
  for now, however, is likely through `config.mk` (what `./configure` emits), so
  the build system currently parses this information.

There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86\_64 and x86, but more portability is always
welcome!

---

Future functionality left to implement

* [ ] Re-verify that multi-host builds work
* [ ] Verify android build works
* [ ] Verify iOS build work (mostly compiler-rt)
* [ ] Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc
* [ ] Implement testing -- this is a huge bullet point with lots of sub-bullets
* [ ] Build and generate documentation (plus the various tools we have in-tree)
* [ ] Move various src/etc scripts into Rust -- not sure how this interacts with `make` build system
* [ ] Implement `make install` - like testing this is also quite massive
* [x] Deduplicate version information with makefiles
@bors

This comment has been minimized.

Copy link
Contributor

commented Feb 11, 2016

💔 Test failed - auto-mac-64-opt

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Feb 11, 2016

@bors: retry

On Thu, Feb 11, 2016 at 2:16 PM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-mac-64-opt
http://buildbot.rust-lang.org/builders/auto-mac-64-opt/builds/8008


Reply to this email directly or view it on GitHub
#31123 (comment).

@bors

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2016

⌛️ Testing commit 55dd595 with merge 78a5d5b...

bors added a commit that referenced this pull request Feb 12, 2016
Auto merge of #31123 - alexcrichton:who-doesnt-want-two-build-systems…
…, r=brson

This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:

1. The `./configure` script is run with `--enable-rustbuild` and other standard
   configuration options.
2. A `Makefile` is generate which proxies commands to the new build system.
3. The new build system has a Python script entry point which manages
   downloading both a Rust and Cargo nightly. This initial script also manages
   building the build system itself (which is written in Rust).
4. The build system, written in rust and called `bootstrap`, architects how to
   call `cargo` and manages building all native libraries and such.

One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:

* The makefiles are impenetrable to all but a few people on this
  planet. This means that contributions to the build system are almost
  nonexistent, and furthermore if a build system change is needed it's
  incredibly difficult to figure out how to do so. This hindrance prevents us
  from doing some "perhaps fancier" things we may wish to do in make.

* Our build system, while portable, is unfortunately not infinitely portable
  everywhere.  For example the recently-introduced MSVC target is quite unlikely
  to have `make` installed by default (e.g. it requires building inside of an
  MSYS2 shell currently). Conversely, the portability of make comes at a cost of
  crazy and weird hacks to work around all sorts of versions of software
  everywhere, especially when it comes to the configure script and makefiles.
  By rewriting this logic in one of the most robust platforms there is, Rust,
  we get to assuage all of these worries for free!

* There's a standard tool to build Rust crates, Cargo, but the standard library
  and compiler don't use it. This means that they cannot benefit easily from the
  crates.io ecosystem, nor can the ecosystem benefit from a standard way to
  build this repository itself. Moving to Cargo should help assuage both of
  these needs. This has the added benefit of making the compiler more
  approachable for newbies as working on the compiler will just happen to be
  working on a large Cargo project, all the same standard tools and tricks will
  apply.

* There's a huge amount of portability information in the main distribution, for
  example around cross compiling, compiling on new OSes, etc. Pushing this logic
  into standard crates (like `gcc`) enables the community to immediately benefit
  from new build logic.

Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:

1. Land a second build system in-tree that can be itereated on an and
   contributed to. This will not be used just yet in terms of gating new commits
   to the repo.
2. Over time, bring the second build system to feature parity with the old build
   system, start setting up CI for both build systems.
3. At some point in the future, switch the default to the new build system, but
   keep the old one around.
4. At some further point in the future, delete the entire old build system.

---

Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the `src/bootstrap`
directory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:

* Compiling LLVM with `cmake` instead of `./configure` + `make`. The LLVM
  project is removing their autotools build system, so we'd have to make this
  transition eventually anyway.

* Compiling compiler-rt with `cmake` as well (for the same rationale as above).

* Adding `Cargo.toml` to map out the dependency graph to all crates, and also
  adding `build.rs` files where appropriate. For example `alloc_jemalloc` has a
  script to build jemalloc, `flate` has a script to build `miniz.c`, `std` will
  build `libbacktrace`, etc.

* Orchestrating all the calls to `cargo` to build the standard distribution,
  following the normal bootstrapping process. This also tracks dependencies
  between steps to ensure cross-compilation targets happen as well.

* Configuration is intended to eventually be done through a `config.toml` file,
  so support is implemented for this. The most likely vector of configuration
  for now, however, is likely through `config.mk` (what `./configure` emits), so
  the build system currently parses this information.

There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86\_64 and x86, but more portability is always
welcome!

---

Future functionality left to implement

* [ ] Re-verify that multi-host builds work
* [ ] Verify android build works
* [ ] Verify iOS build work (mostly compiler-rt)
* [ ] Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc
* [ ] Implement testing -- this is a huge bullet point with lots of sub-bullets
* [ ] Build and generate documentation (plus the various tools we have in-tree)
* [ ] Move various src/etc scripts into Rust -- not sure how this interacts with `make` build system
* [ ] Implement `make install` - like testing this is also quite massive
* [x] Deduplicate version information with makefiles

@bors bors merged commit 55dd595 into rust-lang:master Feb 12, 2016

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details
@aturon

This comment has been minimized.

Copy link
Member

commented Feb 12, 2016

🍻

@alexcrichton alexcrichton deleted the alexcrichton:who-doesnt-want-two-build-systems branch Feb 12, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.