From 3f69ddda3305e564e465b917b3e9a3ba401a0b43 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sun, 3 Jun 2018 22:09:21 -0500 Subject: [PATCH 1/6] add a bit about crater --- src/tests/intro.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index e8cf34aff..7c3832ed9 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -170,7 +170,38 @@ communicate with the server to coordinate running tests (see ## Crater -TODO +[Crater](https://github.com/rust-lang-nursery/crater) is a tool for compiling +and running tests for _every_ crate on [crates.io](https://crates.io/). It is +mainly used for checking for extent of breakage when implementing potentially +breaking changes. + +### When to run Crater + +You should request a crater run if your PR makes large changes to the compiler +or could cause breakage. If you are unsure, feel free to ask your PR's reviewer. + +### Requesting Crater Runs + +The rust team maintains a few machines that can be used for running crater runs +on the changes introduced by a PR. If your PR needs a crater run, leave a +comment for the triage team in the PR thread. Your will be enqueued by the +triage team and the results will be posted when they are ready. A crater run +usually takes a few days (as of this writing). + +While crater is really useful, it is also important to be aware of a few caveats: + +- Not all code is on crates.io! There is a lot of code in repos on GitHub and + elsewhere. Also, companies may not wish to publish their code. Thus, a + successful crater run is not a magically green light that there will be no + breakage; you still need to be careful. + +- Crater only runs Linux builds (on x86_64, I believe). Thus, other + architectures and platforms are not tested. Critically, this includes + Windows. + +- Many crates are not tested. This could be for a lot of reasons, including + that the crate doesn't compile any more (e.g. used old nightly features), + has broken or flaky tests, requires network access, or other reasons. ## Further reading From 4543f7cfd7fc008cfee993623386496dc3232377 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 3 Jun 2018 22:40:29 -0500 Subject: [PATCH 2/6] add more on crater + perf runs --- src/tests/intro.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/tests/intro.md b/src/tests/intro.md index 7c3832ed9..034b7aac1 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -203,6 +203,23 @@ While crater is really useful, it is also important to be aware of a few caveats that the crate doesn't compile any more (e.g. used old nightly features), has broken or flaky tests, requires network access, or other reasons. +- Before crater can be run, `@bors try` needs to suceed. This means that if + your code doesn't compile or fails tests, you cannot run crater. + +## Perf runs + +A lot of work is put into improving the performance of the compiler and +preventing performance regressions. A "perf run" is used to compare the +performance of the compiler in different configurations for a large collection +of popular crates. Different configurations include "fresh builds", builds +with incremental compilation, etc. + +The result of a perf run is a comparison between two versions of the +compiler (by their commit hashes). + +You should request a perf run if your PR may affect performance, especially +if it can affect performance adversely. + ## Further reading The following blog posts may also be of interest: From d12d0e91f4c97b5c061c4a151b050d642138f641 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 3 Jun 2018 22:46:11 -0500 Subject: [PATCH 3/6] Address review feedback --- src/tests/intro.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 034b7aac1..f4164de5e 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -171,9 +171,10 @@ communicate with the server to coordinate running tests (see ## Crater [Crater](https://github.com/rust-lang-nursery/crater) is a tool for compiling -and running tests for _every_ crate on [crates.io](https://crates.io/). It is -mainly used for checking for extent of breakage when implementing potentially -breaking changes. +and running tests for _every_ crate on [crates.io](https://crates.io/) (and a +few on GitHub). It is mainly used for checking for extent of breakage when +implementing potentially breaking changes and ensuring lack of breakage by +running beta vs stable compiler versions. ### When to run Crater @@ -184,9 +185,16 @@ or could cause breakage. If you are unsure, feel free to ask your PR's reviewer. The rust team maintains a few machines that can be used for running crater runs on the changes introduced by a PR. If your PR needs a crater run, leave a -comment for the triage team in the PR thread. Your will be enqueued by the -triage team and the results will be posted when they are ready. A crater run -usually takes a few days (as of this writing). +comment for the triage team in the PR thread. Please inform the team whether +you require a "check-only" crater run, a "build only" crater run, or a +"build-and-test" crater run. The difference is primarily in time; the +conservative (if you're not sure) option is to go for the build-and-test run. +If making changes that will only have an effect at compile-time (e.g., +implementing a new trait) then you only need a check run. + +Your PR will be enqueued by the triage team and the results will be posted when +they are ready. Check runs will take around ~3-4 days, with the other two +taking 5-6 days on average. While crater is really useful, it is also important to be aware of a few caveats: @@ -195,9 +203,8 @@ While crater is really useful, it is also important to be aware of a few caveats successful crater run is not a magically green light that there will be no breakage; you still need to be careful. -- Crater only runs Linux builds (on x86_64, I believe). Thus, other - architectures and platforms are not tested. Critically, this includes - Windows. +- Crater only runs Linux builds on x86_64. Thus, other architectures and + platforms are not tested. Critically, this includes Windows. - Many crates are not tested. This could be for a lot of reasons, including that the crate doesn't compile any more (e.g. used old nightly features), From dcdb6d9607f552238c0740f67d5f56929d555f82 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 3 Jun 2018 22:48:03 -0500 Subject: [PATCH 4/6] long line --- src/tests/intro.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index f4164de5e..79aa0fd6a 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -196,7 +196,8 @@ Your PR will be enqueued by the triage team and the results will be posted when they are ready. Check runs will take around ~3-4 days, with the other two taking 5-6 days on average. -While crater is really useful, it is also important to be aware of a few caveats: +While crater is really useful, it is also important to be aware of a few +caveats: - Not all code is on crates.io! There is a lot of code in repos on GitHub and elsewhere. Also, companies may not wish to publish their code. Thus, a From a2bc876a274f2d2bf131a8feca23a3ea95649dd7 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 19:23:19 -0500 Subject: [PATCH 5/6] tests dont need to pass --- src/tests/intro.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 79aa0fd6a..bd8a60a88 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -211,8 +211,8 @@ caveats: that the crate doesn't compile any more (e.g. used old nightly features), has broken or flaky tests, requires network access, or other reasons. -- Before crater can be run, `@bors try` needs to suceed. This means that if - your code doesn't compile or fails tests, you cannot run crater. +- Before crater can be run, `@bors try` needs to suceed in building artifacts. + This means that if your code doesn't compile, you cannot run crater. ## Perf runs From 641a06f94c365ea0a28c4f0c6d0ce943e727fcfd Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 19:46:05 -0500 Subject: [PATCH 6/6] fix typo --- src/tests/intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index bd8a60a88..bc84c9721 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -211,7 +211,7 @@ caveats: that the crate doesn't compile any more (e.g. used old nightly features), has broken or flaky tests, requires network access, or other reasons. -- Before crater can be run, `@bors try` needs to suceed in building artifacts. +- Before crater can be run, `@bors try` needs to succeed in building artifacts. This means that if your code doesn't compile, you cannot run crater. ## Perf runs