Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way to track Rustfix UI test coverage #59398

Merged
merged 4 commits into from Mar 29, 2019
Merged

Conversation

@phansch
Copy link
Member

@phansch phansch commented Mar 24, 2019

This came out of the first Rustfix WG meeting.

One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with MachineApplicable suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing // run-rustfix headers.

This PR adds a --rustfix-coverage flag to ./x.py and compiletest to
list the files with the missing headers in /tmp/rustfix_missing_coverage.txt.

From that file we can create the tracking issue and at some point also
enforce the // run-rustfix flag on UI tests with MachineApplicable
lints.

@rust-highfive
Copy link
Collaborator

@rust-highfive rust-highfive commented Mar 24, 2019

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

@phansch
Copy link
Member Author

@phansch phansch commented Mar 24, 2019

r? @oli-obk

and cc @killercup I guess? =)

@phansch phansch force-pushed the rustfix_coverage branch from e2b4d1e to 1e18c0e Mar 24, 2019
@rust-highfive

This comment has been hidden.

This came out of the first Rustfix WG meeting.

One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with `MachineApplicable` suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing `// run-rustfix` headers.

This PR adds a `--rustfix-coverage` flag to `./x.py` and compiletest to
list the files with the missing headers in `/tmp/rustfix_missing_coverage.txt`.

From that file we can create the tracking issue and at some point also
enforce the `// run-rustfix` flag on UI tests with `MachineApplicable`
lints.
@rust-highfive
Copy link
Collaborator

@rust-highfive rust-highfive commented Mar 24, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0d355238:start=1553443553537255520,finish=1553443554484076371,duration=946820851
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:21:45] 
[01:21:45] running 9 tests
[01:21:45] iiiiiiiii
[01:21:45] 
[01:21:45]  finished in 0.158
[01:21:45] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:22:01] 
[01:22:01] running 120 tests
[01:22:27] .iiiii...i.....i..i...i..i.i...i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i.i 100/120
[01:22:32] .i......iii.i.....ii
[01:22:32] 
[01:22:32]  finished in 30.954
[01:22:32] travis_fold:end:test_debuginfo

---
[01:51:51]    Compiling ansi_term v0.11.0
[01:51:51]    Compiling difference v2.0.0
[01:51:52]    Compiling pretty_assertions v0.5.1
[01:51:52]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[01:51:56] error[E0063]: missing field `rustfix_coverage` in initializer of `flags::Subcommand`
[01:51:56]      |
[01:51:56] 1851 |         config.cmd = Subcommand::Test {
[01:51:56] 1851 |         config.cmd = Subcommand::Test {
[01:51:56]      |                      ^^^^^^^^^^^^^^^^ missing `rustfix_coverage`
[01:51:56] 
[01:51:56] error[E0063]: missing field `rustfix_coverage` in initializer of `flags::Subcommand`
[01:51:56]      |
[01:51:56] 1892 |         config.cmd = Subcommand::Test {
[01:51:56] 1892 |         config.cmd = Subcommand::Test {
[01:51:56]      |                      ^^^^^^^^^^^^^^^^ missing `rustfix_coverage`
[01:51:56] error: aborting due to 2 previous errors
[01:51:56] 
[01:51:56] For more information about this error, try `rustc --explain E0063`.
[01:51:56] error: Could not compile `bootstrap`.
---
[01:51:56] 
[01:51:56] 
[01:51:56] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:51:56] Build completed unsuccessfully in 0:41:59
[01:51:56] make: *** [check] Error 1
[01:51:56] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:2938cf98
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Mar 24 17:58:01 UTC 2019
---
43088 ./obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnutravis_time:end:13bc5fdf:start=1553450283476555580,finish=1553450283533642021,duration=57086441
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0925885e
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:10e0a26c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

phansch added 2 commits Mar 25, 2019
Not all suggestions come from lints
@phansch
Copy link
Member Author

@phansch phansch commented Mar 25, 2019

This will need a bump of the rustfix version as well, once rust-lang/rustfix#165 is merged and released.

@oli-obk
Copy link
Contributor

@oli-obk oli-obk commented Mar 25, 2019

I don't think we should be writing to /tmp. Can you fetch the path that resolves to build in the default configuration (or is build hardcoded?) and dump the file in there?

@phansch
Copy link
Member Author

@phansch phansch commented Mar 25, 2019

I don't think we should be writing to /tmp

Good point, fixed in d808bd4. It's now written to ./build/<host-triple>/test/ui/rustfix_missing_coverage.txt.

@oli-obk
Copy link
Contributor

@oli-obk oli-obk commented Mar 25, 2019

@bors r+

now you've got me thinking about how I can get our CI to emit this metric on every build so I can create a graph

@bors
Copy link
Contributor

@bors bors commented Mar 25, 2019

📌 Commit d808bd4 has been approved by oli-obk

@phansch
Copy link
Member Author

@phansch phansch commented Mar 26, 2019

I think this won't succeed yet because it will still crash on some UI tests until rust-lang/rustfix#165 is merged =)

@oli-obk
Copy link
Contributor

@oli-obk oli-obk commented Mar 26, 2019

Since we're not running the script in CI, is that problematic? I mean until it's merged and rustfix is updated in this repo, we can just remove the broken tests locally.

@phansch
Copy link
Member Author

@phansch phansch commented Mar 26, 2019

Since we're not running the script in CI, is that problematic?

Oh, you're completely right! It will be fine then =)

phansch added a commit to phansch/compiletest-rs that referenced this issue Mar 26, 2019
Pretty much a copy of rust-lang/rust#59398 minus
the rustc bootstrap things.

This will allow rustfix coverage tracking in Clippy and other
libraries that emit `MachineApplicable` diagnostics.
Centril added a commit to Centril/rust that referenced this issue Mar 28, 2019
Add a way to track Rustfix UI test coverage

This came out of the first Rustfix WG meeting.

One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with `MachineApplicable` suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing `// run-rustfix` headers.

This PR adds a `--rustfix-coverage` flag to `./x.py` and compiletest to
list the files with the missing headers in `/tmp/rustfix_missing_coverage.txt`.

From that file we can create the tracking issue and at some point also
enforce the `// run-rustfix` flag on UI tests with `MachineApplicable`
lints.
bors added a commit that referenced this issue Mar 28, 2019
Rollup of 10 pull requests

Successful merges:

 - #58717 (Add FromStr impl for NonZero types)
 - #59091 (Combine input and eval_always query types)
 - #59216 (Type dependent defs wrappers)
 - #59318 (rustc: Update linker flavor inference from filename)
 - #59320 (rustc: Allow using `clang` for wasm32 targets)
 - #59363 (#59361 Moved rustc edition opt to short list)
 - #59371 (ffi: rename VaList::copy to VaList::with_copy)
 - #59398 (Add a way to track Rustfix UI test coverage)
 - #59408 (compiletest: make path normalization smarter)
 - #59429 (When moving out of a for loop head, suggest borrowing it in nll mode)

Failed merges:

r? @ghost
bors added a commit that referenced this issue Mar 28, 2019
Rollup of 10 pull requests

Successful merges:

 - #58717 (Add FromStr impl for NonZero types)
 - #59091 (Combine input and eval_always query types)
 - #59216 (Type dependent defs wrappers)
 - #59318 (rustc: Update linker flavor inference from filename)
 - #59320 (rustc: Allow using `clang` for wasm32 targets)
 - #59363 (#59361 Moved rustc edition opt to short list)
 - #59371 (ffi: rename VaList::copy to VaList::with_copy)
 - #59398 (Add a way to track Rustfix UI test coverage)
 - #59408 (compiletest: make path normalization smarter)
 - #59429 (When moving out of a for loop head, suggest borrowing it in nll mode)

Failed merges:

r? @ghost
bors added a commit that referenced this issue Mar 28, 2019
Rollup of 10 pull requests

Successful merges:

 - #58717 (Add FromStr impl for NonZero types)
 - #59091 (Combine input and eval_always query types)
 - #59216 (Type dependent defs wrappers)
 - #59318 (rustc: Update linker flavor inference from filename)
 - #59320 (rustc: Allow using `clang` for wasm32 targets)
 - #59363 (#59361 Moved rustc edition opt to short list)
 - #59371 (ffi: rename VaList::copy to VaList::with_copy)
 - #59398 (Add a way to track Rustfix UI test coverage)
 - #59408 (compiletest: make path normalization smarter)
 - #59429 (When moving out of a for loop head, suggest borrowing it in nll mode)

Failed merges:

r? @ghost
@bors bors merged commit d808bd4 into rust-lang:master Mar 29, 2019
1 check passed
@phansch phansch deleted the rustfix_coverage branch Apr 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants