mk: Switch rustbuild to the default build system #37817

Merged
merged 1 commit into from Dec 7, 2016
@alexcrichton
Member
alexcrichton commented Nov 16, 2016 edited

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on internals which slates deletion of the makefiles on
2017-02-02.

This commit also updates various documentation in README.md,
CONTRIBUTING.md, src/bootstrap/README.md, and throughout the source code of
rustbuild itself.

@rust-highfive
Collaborator

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

@alexcrichton
Member

r? @brson

Note that this shouldn't be r+'d yet as I'd like to wait for appveyor/Travis to go green (trying to run all the builders there).

Also if there's any more requests for documentation anywhere, I'd be more than happy to oblige!

@rust-highfive rust-highfive assigned brson and unassigned aturon Nov 16, 2016
+To learn more about the driver and top-level targets, you can execute:
+
+```sh
+python x.py --help
@petrochenkov
petrochenkov Nov 16, 2016 Contributor

I run x.py --help and it starts downloading something big from the internet instead of showing help.
I don't think this is an appropriate behavior.

@badboy
badboy Nov 17, 2016 Contributor

Tracked here: #37305

@petrochenkov

Fix #37305 before merging

+of the compiler. You can execute it as:
+
+```sh
+python x.py build
@petrochenkov
petrochenkov Nov 17, 2016 edited Contributor

Is bootstrap python3-compatible, btw? In case python defaults to 3. This is probably worth mentioning.
(Or build fails in some other way if python == python3, but python2 is still available? I don't remember exactly.)

@xen0n
xen0n Nov 17, 2016 Contributor

Yes all the Python code in Rust repo is Py3k-compatible AFAIK, which is not so much btw. The whole Python 2-only thing in README is outdated actually.

@retep998
retep998 Nov 17, 2016 Member

Last I heard, Python 2 was only necessary for LLVM.

@xen0n
xen0n Nov 17, 2016 Contributor

@retep998 Yeah you're right, there is this which means precisely Python 2.7.x is required, but rustbuild itself isn't limited to that. Which means compilation will only fail on systems with no Python 2.x at all due to LLVM, but should be fine otherwise.

@alexcrichton
alexcrichton Nov 17, 2016 Member

AFAIK bootstrap.py is fully compatible, and then later version detection happens in rustbuild itself.

configure
+ warn "the makefile-based build system is deprecated in favor of rustbuild"
+ msg ""
+ msg "It is recommended you avoid passing --disable-rustbuild to get your"
+ msg "build working as the makefiles will be deleted on 2016-01-02. If you"
@petrochenkov
petrochenkov Nov 17, 2016 Contributor

Sorry for a silly question, but is the date in standard format (2 January) or in Freedom Units (1 February)?.

@cuviper
cuviper Nov 17, 2016 Contributor

I just commented on the internals thread -- I think it should actually be February 2nd.

@rkruppe
rkruppe Nov 17, 2016 Contributor

In any case, the year is wrong. (In the PR description, too)

src/bootstrap/lib.rs
+//! to execute each in sequence. Each time rustbuild is invoked, it will simply
+//! iterate through this list of steps and execute each serially in turn.
+//! Rustbuild relies in each step internally being incremental and also
+//! internally parallelizing.
@petrochenkov
petrochenkov Nov 17, 2016 Contributor

Could you document explicit parallelizing with -jN somewhere as well?

configure
+ warn "the makefile-based build system is deprecated in favor of rustbuild"
+ msg ""
+ msg "It is recommended you avoid passing --disable-rustbuild to get your"
+ msg "build working as the makefiles will be deleted on 2016-02-02. If you"
@cuviper
cuviper Nov 17, 2016 Contributor

*2017

CONTRIBUTING.md
+python x.py build
+```
+
+If you're on a Unix-based system, you can also use the shorter version:
@retep998
retep998 Nov 17, 2016 Member

Technically you can also use a different short version on Windows if you have .py files associated with Python. x.py build in cmd and .\x.py build in powershell.

@petrochenkov
petrochenkov Nov 17, 2016 Contributor

And MSYS2 is just "Unix-based" enough to see the shebang :D

README.md
```sh
-$ ./configure
-$ make && make install
+$ python x.py build
@retep998
retep998 Nov 17, 2016 Member

This prompt is supposed to be cmd.exe according to the above paragraph, so using $ is incorrect here.

CONTRIBUTING.md
-questions on the build system internals, try asking in
-[`#rust-internals`][pound-rust-internals].
+The build system lives in [the `src/bootstrap` directory][bootstrap] in the
+project root. Our build system its itself written in Rust and is based on Cargo
@pitdicker
pitdicker Nov 17, 2016 Contributor

*is itself

src/bootstrap/lib.rs
+//!
+//! ## Architecture
+//!
+//! Although this build system defers most of the compilcated logic to Cargo
@pitdicker
pitdicker Nov 17, 2016 Contributor

*complicated

@fbstj
fbstj Nov 17, 2016

tho it is also pretty compile-cated

src/bootstrap/step.rs
+ // That logic is all defined in the `native` module so we just delegate to
+ // the relevant function there. The argument to the closure passed to `run`
+ // is a `Step` (defined below) which encapsulates information like the
+ // stage, host, target, etc.
@pitdicker
pitdicker Nov 17, 2016 edited Contributor

*compiled (I guess) line 77

src/bootstrap/lib.rs
+//!
+//! Although this build system defers most of the compilcated logic to Cargo
+//! itself, it still needs to maintain a list of targets and dependencies which
+//! it can itself perform. Rustbuild is comprised of of a list of rules with
@pitdicker
pitdicker Nov 17, 2016 Contributor

*perform on. Rustbuild is made up of

Just a suggestion 'of of' reads a bit difficult

src/bootstrap/lib.rs
+//! to execute each in sequence. Each time rustbuild is invoked, it will simply
+//! iterate through this list of steps and execute each serially in turn.
+//! Rustbuild relies in each step internally being incremental and also
+//! internally parallelizing.
@pitdicker
pitdicker Nov 17, 2016 Contributor

*For each step Rustbuild relies on the step to be internally incremental and also to be internally parallelizing.

@alexcrichton
Member

@petrochenkov can you elaborate on why you think #37305 is a blocker for landing this PR? I unfortunately don't see any path forward to solving it, so any opinions on how to solve it would also be greatly appreciated!

@bors
Contributor
bors commented Nov 26, 2016

☔️ The latest upstream changes (presumably #38008) made this pull request unmergeable. Please resolve the merge conflicts.

@rkruppe rkruppe added a commit to rkruppe/rust that referenced this pull request Nov 28, 2016
@rkruppe rkruppe Define VISIBILITY_HIDDEN when compiling compiler-rt
This is cherry-picked from #37817 which seems to be stalled and currently needs to be rebased anyway.

r? @alexcrichton (who authored this change)
02249e8
@alexcrichton
Member

Ok I got as successful a run as we're gonna get, so I think this is good to go.

r? @brson

@bors bors added a commit that referenced this pull request Nov 29, 2016
@bors bors Auto merge of #38046 - rkruppe:fix-32bit-rustbuild, r=alexcrichton
Fix rustbuild on 32 bit Linux

This is cherry-picked from #37817 which seems to be stalled and currently needs to be rebased anyway.

r? @alexcrichton (who authored this change)
5de15be
README.md
- > install locations can be adjusted by passing a `--prefix` argument
+ > ***Note:*** Currently `sudo make install` is not supported, so if you
+ > do not have permission to modify the destination directory then you'll
+ > want to install to a local directory like `$HOME/rust`.
@brson
brson Nov 29, 2016 Contributor

Why isn't sudo make install supported? That is the command one typically needs to install Rust from source.

@brson
brson Nov 29, 2016 Contributor

Hm, I guess I'm not sure how one does a root install today, but I assume it's with sudo. Perhaps the makefiles were doing clever things to not spew root-owned files all over the build directory.

@alexcrichton
alexcrichton Nov 29, 2016 Member

Ah so I'm less concerned about the build directory today (which we will hose over with root owned files if you sudo make install in a fresh dir). What I'm worried about here is that Cargo access global state, like your home directory, where a sudo make install could damage quite a bit.

So basically tl;dr; is that sudo cargo build is likely to go terribly wrong (although I haven't tried to hard to verify this). Transitively this would imply that sudo make install would also go terribly wrong as it involves a cargo build along the way.

I wonder if though we could just detect this in rustbuild and avoid running Cargo at all...

appveyor.yml
@@ -4,21 +4,21 @@ environment:
- MSYS_BITS: 64
TARGET: x86_64-pc-windows-msvc
CHECK: check
- CONFIGURE_ARGS: --enable-llvm-assertions --enable-debug-assertions
+ CONFIGURE_ARGS: --enable-vendor
@brson
brson Nov 29, 2016 Contributor

Why are all these configurations remove assertions? And why isn't this patch adding --enable-vendor to the travis configs?

@alexcrichton
alexcrichton Nov 29, 2016 Member

Oh this is where the windows/travis builds differ ever so slightly. The Travis builds go through src/ci/run.sh which unconditionally adds --enable-vendor. The AppVeyor builds, however, do configure + make manually (avoiding src/ci/run.sh) which means we have to duplicate a bit of logic, and is why --enable-vendor is explicitly spelled out here.

I can look into unifying these two paths, though.

appveyor.yml
@@ -91,6 +92,8 @@ install:
test_script:
- sh ./configure
+ --enable-llvm-assertions
+ --enable-debug-assertions
@brson
brson Nov 29, 2016 Contributor

Well that answers one question.

@brson brson dismissed their review Nov 29, 2016

oops

src/rustllvm/llvm-auto-clean-trigger
@@ -1,4 +1,4 @@
# If this file is modified, then llvm will be forcibly cleaned and then rebuilt.
# The actual contents of this file do not matter, but to trigger a change on the
# build bots then the contents should be changed so git updates the mtime.
-2016-10-29
+2016-11-17
@brson
brson Nov 29, 2016 Contributor

Why does this PR update LLVM?

@alexcrichton
alexcrichton Nov 29, 2016 Member

Oh I was just using this to bust the caches to ensure we rebuild LLVM from scratch on all the bots , this doesn't actually update LLVM though.

@brson
Contributor
brson commented Nov 29, 2016

@alexcrichton this looks fantastic. The comments are crystal clear. I mostly just had questions, but I am concerned about sudo make install not working. That seems like something people will hit.

@alexcrichton
Member

Ok, discussed offline with @brson, and the sudo business is now taken care of by having sudo imply --frozen, which means that Cargo won't touch anything in HOME

@bors
Contributor
bors commented Nov 30, 2016

☔️ The latest upstream changes (presumably #37800) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 1, 2016

📌 Commit b6a6f11 has been approved by brson

@frewsxcv
Member
frewsxcv commented Dec 1, 2016

Looks like Travis is failing, though it's not clear to me why.

@alexcrichton
Member

I've seen that occasionally and maybe this is travis-ci/travis-ci#4192? Just a spurious OOM?

@bors
Contributor
bors commented Dec 1, 2016

⌛️ Testing commit b6a6f11 with merge c9dd02b...

@bors bors added a commit that referenced this pull request Dec 1, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
c9dd02b
@bors
Contributor
bors commented Dec 1, 2016

💔 Test failed - auto-linux-musl-64-opt

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 1, 2016

📌 Commit 56fa9cf has been approved by brson

@bors
Contributor
bors commented Dec 1, 2016

⌛️ Testing commit 56fa9cf with merge 00c1db2...

@bors bors added a commit that referenced this pull request Dec 1, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
00c1db2
@bors
Contributor
bors commented Dec 1, 2016

💔 Test failed - auto-linux-cross-opt

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 1, 2016

📌 Commit 8267d37 has been approved by brson

@bors
Contributor
bors commented Dec 1, 2016

⌛️ Testing commit 8267d37 with merge 34e6fb9...

@bors bors added a commit that referenced this pull request Dec 1, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
34e6fb9
@bors
Contributor
bors commented Dec 2, 2016

💔 Test failed - auto-linux-musl-64-opt

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 2, 2016

📌 Commit 95d3831 has been approved by brson

@bors
Contributor
bors commented Dec 2, 2016

⌛️ Testing commit 95d3831 with merge 84bd640...

@bors bors added a commit that referenced this pull request Dec 2, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
84bd640
@bors
Contributor
bors commented Dec 2, 2016

💔 Test failed - auto-linux-64-x-android-t

@bors
Contributor
bors commented Dec 2, 2016

☔️ The latest upstream changes (presumably #38050) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 2, 2016

📌 Commit 14d05c8 has been approved by brson

@bors
Contributor
bors commented Dec 2, 2016

⌛️ Testing commit 14d05c8 with merge 4fcd8ae...

@bors bors added a commit that referenced this pull request Dec 2, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
4fcd8ae
@bors
Contributor
bors commented Dec 2, 2016

💔 Test failed - auto-win-msvc-32-opt

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 2, 2016

📌 Commit 8765f6e has been approved by brson

@bors
Contributor
bors commented Dec 3, 2016

⌛️ Testing commit 8765f6e with merge 702954f...

@bors bors added a commit that referenced this pull request Dec 3, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
702954f
@bors
Contributor
bors commented Dec 3, 2016

💔 Test failed - auto-win-msvc-32-opt

@alexcrichton
Member

@bors: r+

@bors
Contributor
bors commented Dec 3, 2016

📌 Commit 76f9e9e has been approved by alexcrichton

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 3, 2016

💡 This pull request was already approved, no need to approve it again.

  • There's another pull request that is currently being tested, blocking this pull request: #38148
@bors
Contributor
bors commented Dec 3, 2016

📌 Commit 76f9e9e has been approved by brson

@bors
Contributor
bors commented Dec 4, 2016

⌛️ Testing commit 76f9e9e with merge 6918d04...

@bors bors added a commit that referenced this pull request Dec 4, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
6918d04
@bors
Contributor
bors commented Dec 4, 2016

💔 Test failed - auto-linux-musl-64-opt

@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 5, 2016

📌 Commit 4755f95 has been approved by brson

@bors
Contributor
bors commented Dec 5, 2016

⌛️ Testing commit 4755f95 with merge 0ae2037...

@bors bors added a commit that referenced this pull request Dec 5, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
0ae2037
@bors
Contributor
bors commented Dec 5, 2016

💔 Test failed - auto-linux-64-x-android-t

@bors
Contributor
bors commented Dec 5, 2016

☔️ The latest upstream changes (presumably #38100) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton alexcrichton mk: Switch rustbuild to the default build system
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.

Closes #37858
0e272de
@alexcrichton
Member

@bors: r=brson

@bors
Contributor
bors commented Dec 7, 2016

📌 Commit 0e272de has been approved by brson

@bors
Contributor
bors commented Dec 7, 2016

⌛️ Testing commit 0e272de with merge 7846610...

@bors bors added a commit that referenced this pull request Dec 7, 2016
@bors bors Auto merge of #37817 - alexcrichton:rustbuild-default, r=brson
mk: Switch rustbuild to the default build system

This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2017-02-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
7846610
@bors bors merged commit 0e272de into rust-lang:master Dec 7, 2016

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build failed
Details
homu Test successful
Details
@frewsxcv
Member
frewsxcv commented Dec 8, 2016

It looks like Travis started failing everything after this pull request merged.

@stjepang stjepang added a commit to stjepang/rust that referenced this pull request Dec 16, 2016
@rkruppe @stjepang rkruppe + stjepang Define VISIBILITY_HIDDEN when compiling compiler-rt
This is cherry-picked from #37817 which seems to be stalled and currently needs to be rebased anyway.

r? @alexcrichton (who authored this change)
248f9f8
@alexcrichton alexcrichton deleted the alexcrichton:rustbuild-default branch Dec 19, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment