Skip to content

compiletest: Report filtered and --skip-ed tests in metrics #155555

Open
jyn514 wants to merge 2 commits intorust-lang:mainfrom
ferrocene:jyn/compiletest-integration-tests
Open

compiletest: Report filtered and --skip-ed tests in metrics #155555
jyn514 wants to merge 2 commits intorust-lang:mainfrom
ferrocene:jyn/compiletest-integration-tests

Conversation

@jyn514
Copy link
Copy Markdown
Member

@jyn514 jyn514 commented Apr 20, 2026

Ferrocene depends on these metrics for our certification. For various reasons our CI invokes bootstrap with --skip flags, so we want to report ignored tests even if they're ignored manually with --skip.

To make sure this doesn't regress, add an integration test suite for compiletest itself. This can't fit into the existing compiletest-self-test suite because it's testing the interface between bootstrap and compiletest, not just whether the tests pass or fail. Hopefully this will come in handy for more things in the future.


Changes in this PR:

  • Move deserialization of compiletest JSON from bootstrap to build_helper. Derive Debug on all its types.
  • Add std to the sysroot before running compiletest self-tests, so that it can run integration tests and not just unit tests.
  • Add a new FilteredOut metric kind. Handle it appropriately (by not doing anything) in progress reporting, but save it to the metrics.json file.

@rustbot
Copy link
Copy Markdown
Collaborator

rustbot commented Apr 20, 2026

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Some changes occurred in src/tools/compiletest

cc @jieyouxu

@rustbot rustbot added A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 20, 2026
@rustbot
Copy link
Copy Markdown
Collaborator

rustbot commented Apr 20, 2026

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: @jieyouxu, @oli-obk, @wesleywiser, bootstrap
  • @jieyouxu, @oli-obk, @wesleywiser, bootstrap expanded to 8 candidates
  • Random selection from Mark-Simulacrum, clubby789, jieyouxu, wesleywiser

@rust-log-analyzer

This comment has been minimized.

@jyn514 jyn514 force-pushed the jyn/compiletest-integration-tests branch 2 times, most recently from 2235bed to c4f30fb Compare April 20, 2026 14:19
@rustbot rustbot added A-CI Area: Our Github Actions CI T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Apr 20, 2026
@rust-log-analyzer

This comment has been minimized.

@jyn514 jyn514 force-pushed the jyn/compiletest-integration-tests branch from c4f30fb to a04daaf Compare April 20, 2026 14:41
@rust-log-analyzer

This comment has been minimized.

@jyn514 jyn514 force-pushed the jyn/compiletest-integration-tests branch from a04daaf to b35ca65 Compare April 20, 2026 15:17
@rust-log-analyzer
Copy link
Copy Markdown
Collaborator

The job aarch64-gnu-llvm-21-2 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)
##[endgroup]
Executing "/scripts/stage_2_test_set2.sh"
+ /scripts/stage_2_test_set2.sh
+ '[' 1 == 1 ']'
+ echo 'PR_CI_JOB set; skipping tidy'
+ SKIP_TIDY='--skip tidy'
+ ../x.py --stage 2 test --skip tidy --skip tests --skip coverage-map --skip coverage-run --skip library --skip tidyselftest
PR_CI_JOB set; skipping tidy
##[group]Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
---
────────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬───────────────────────────────────────────────────────────────────
    0     0 │ [build] llvm <host>
    1     1 │ [build] rustc 0 <host> -> rustc 1 <host>
    2       │-[build] rustdoc 0 <host>
          2 │+[build] rustdoc 0 <host>
          3 │+[build] rustc 1 <host> -> std 1 <host>
────────────┴───────────────────────────────────────────────────────────────────
Stopped on the first failure. Run `cargo insta test` to run all snapshots.

thread 'core::builder::tests::snapshot::test_compiletest_self_test' (11474) panicked at /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.43.1/src/runtime.rs:679:13:
snapshot assertion for 'compiletest_self_test' failed in line 2161
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/ef0fb8a2563200e322fa4419f09f65a63742038c/library/std/src/panicking.rs:689:5
   1: core::panicking::panic_fmt
             at /rustc/ef0fb8a2563200e322fa4419f09f65a63742038c/library/core/src/panicking.rs:80:14
   2: <insta::runtime::SnapshotAssertionContext>::finalize
             at /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.43.1/src/runtime.rs:679:13
   3: insta::runtime::assert_snapshot
             at /cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.43.1/src/runtime.rs:848:13
   4: bootstrap::core::builder::tests::snapshot::test_compiletest_self_test
             at ./src/core/builder/tests.rs:2161:9
   5: bootstrap::core::builder::tests::snapshot::test_compiletest_self_test::{closure#0}
             at ./src/core/builder/tests.rs:2158:36
   6: <bootstrap::core::builder::tests::snapshot::test_compiletest_self_test::{closure#0} as core::ops::function::FnOnce<()>>::call_once

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-CI Area: Our Github Actions CI A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants