Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error building main crate and error building its tests are not differentiated #558

Open
SNCPlay42 opened this issue Jan 13, 2021 · 0 comments

Comments

@SNCPlay42
Copy link

If the old compiler successfully built a crate, but failed to build its tests, and the new compiler failed to build the main crate, that should be reported as a regression (and vice-versa as a fix).

But both failing to build the main crate and failure to build the tests currently result in a BuildFail status (TestFail only comes from failure to run the tests):

crater/src/runner/test.rs

Lines 230 to 280 in 674e4c2

fn build<DB: WriteResults>(
ctx: &TaskCtx<DB>,
build_env: &Build,
local_packages_id: &HashSet<PackageId>,
) -> Fallible<()> {
run_cargo(
ctx,
build_env,
&["build", "--frozen", "--message-format=json"],
true,
local_packages_id,
)?;
run_cargo(
ctx,
build_env,
&["test", "--frozen", "--no-run", "--message-format=json"],
true,
local_packages_id,
)?;
Ok(())
}
fn test<DB: WriteResults>(ctx: &TaskCtx<DB>, build_env: &Build) -> Fallible<()> {
run_cargo(
ctx,
build_env,
&["test", "--frozen"],
false,
&HashSet::new(),
)
}
pub(super) fn test_build_and_test<DB: WriteResults>(
ctx: &TaskCtx<DB>,
build_env: &Build,
local_packages_id: &HashSet<PackageId>,
) -> Fallible<TestResult> {
let build_r = build(ctx, build_env, local_packages_id);
let test_r = if build_r.is_ok() {
Some(test(ctx, build_env))
} else {
None
};
Ok(match (build_r, test_r) {
(Err(err), None) => TestResult::BuildFail(failure_reason(&err)),
(Ok(_), Some(Err(err))) => TestResult::TestFail(failure_reason(&err)),
(Ok(_), Some(Ok(_))) => TestResult::TestPass,
(_, _) => unreachable!(),
})
}

Noticed in rust-lang/rust#80953 (in that case, the new compiler failed the same way as the old one, but if the new compiler did have the regression, it wouldn't have been noticed due to this).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant