Add a way to track Rustfix UI test coverage #59398

phansch · 2019-03-24T13:50:54Z

This came out of the first Rustfix WG meeting.

One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with MachineApplicable suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing // run-rustfix headers.

This PR adds a --rustfix-coverage flag to ./x.py and compiletest to
list the files with the missing headers in /tmp/rustfix_missing_coverage.txt.

From that file we can create the tracking issue and at some point also
enforce the // run-rustfix flag on UI tests with MachineApplicable
lints.

rust-highfive · 2019-03-24T13:51:04Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

phansch · 2019-03-24T13:52:49Z

r? @oli-obk

and cc @killercup I guess? =)

This came out of the first Rustfix WG meeting. One of the goals is to enable Rustfix tests for all UI tests that trigger lints with `MachineApplicable` suggestions. In order to do that we first want to create a tracking issue that lists all files with missing `// run-rustfix` headers. This PR adds a `--rustfix-coverage` flag to `./x.py` and compiletest to list the files with the missing headers in `/tmp/rustfix_missing_coverage.txt`. From that file we can create the tracking issue and at some point also enforce the `// run-rustfix` flag on UI tests with `MachineApplicable` lints.

rust-highfive · 2019-03-24T17:58:07Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:0d355238:start=1553443553537255520,finish=1553443554484076371,duration=946820851
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:21:45] 
[01:21:45] running 9 tests
[01:21:45] iiiiiiiii
[01:21:45] 
[01:21:45]  finished in 0.158
[01:21:45] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:22:01] 
[01:22:01] running 120 tests
[01:22:27] .iiiii...i.....i..i...i..i.i...i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i.i 100/120
[01:22:32] .i......iii.i.....ii
[01:22:32] 
[01:22:32]  finished in 30.954
[01:22:32] travis_fold:end:test_debuginfo

---
[01:51:51]    Compiling ansi_term v0.11.0
[01:51:51]    Compiling difference v2.0.0
[01:51:52]    Compiling pretty_assertions v0.5.1
[01:51:52]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[01:51:56] error[E0063]: missing field `rustfix_coverage` in initializer of `flags::Subcommand`
[01:51:56]      |
[01:51:56] 1851 |         config.cmd = Subcommand::Test {
[01:51:56] 1851 |         config.cmd = Subcommand::Test {
[01:51:56]      |                      ^^^^^^^^^^^^^^^^ missing `rustfix_coverage`
[01:51:56] 
[01:51:56] error[E0063]: missing field `rustfix_coverage` in initializer of `flags::Subcommand`
[01:51:56]      |
[01:51:56] 1892 |         config.cmd = Subcommand::Test {
[01:51:56] 1892 |         config.cmd = Subcommand::Test {
[01:51:56]      |                      ^^^^^^^^^^^^^^^^ missing `rustfix_coverage`
[01:51:56] error: aborting due to 2 previous errors
[01:51:56] 
[01:51:56] For more information about this error, try `rustc --explain E0063`.
[01:51:56] error: Could not compile `bootstrap`.
---
[01:51:56] 
[01:51:56] 
[01:51:56] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:51:56] Build completed unsuccessfully in 0:41:59
[01:51:56] make: *** [check] Error 1
[01:51:56] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:2938cf98
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Mar 24 17:58:01 UTC 2019
---
43088 ./obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnutravis_time:end:13bc5fdf:start=1553450283476555580,finish=1553450283533642021,duration=57086441
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0925885e
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:10e0a26c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

src/tools/compiletest/src/common.rs

Not all suggestions come from lints

phansch · 2019-03-25T06:10:39Z

This will need a bump of the rustfix version as well, once rust-lang/rustfix#165 is merged and released.

oli-obk · 2019-03-25T09:32:59Z

I don't think we should be writing to /tmp. Can you fetch the path that resolves to build in the default configuration (or is build hardcoded?) and dump the file in there?

phansch · 2019-03-25T21:56:58Z

I don't think we should be writing to /tmp

Good point, fixed in d808bd4. It's now written to ./build/<host-triple>/test/ui/rustfix_missing_coverage.txt.

oli-obk · 2019-03-25T23:46:15Z

@bors r+

now you've got me thinking about how I can get our CI to emit this metric on every build so I can create a graph

bors · 2019-03-25T23:46:16Z

📌 Commit d808bd4 has been approved by oli-obk

phansch · 2019-03-26T05:53:27Z

I think this won't succeed yet because it will still crash on some UI tests until rust-lang/rustfix#165 is merged =)

oli-obk · 2019-03-26T07:00:23Z

Since we're not running the script in CI, is that problematic? I mean until it's merged and rustfix is updated in this repo, we can just remove the broken tests locally.

phansch · 2019-03-26T08:15:07Z

Since we're not running the script in CI, is that problematic?

Oh, you're completely right! It will be fine then =)

Pretty much a copy of rust-lang/rust#59398 minus the rustc bootstrap things. This will allow rustfix coverage tracking in Clippy and other libraries that emit `MachineApplicable` diagnostics.

Add a way to track Rustfix UI test coverage This came out of the first Rustfix WG meeting. One of the goals is to enable Rustfix tests for all UI tests that trigger lints with `MachineApplicable` suggestions. In order to do that we first want to create a tracking issue that lists all files with missing `// run-rustfix` headers. This PR adds a `--rustfix-coverage` flag to `./x.py` and compiletest to list the files with the missing headers in `/tmp/rustfix_missing_coverage.txt`. From that file we can create the tracking issue and at some point also enforce the `// run-rustfix` flag on UI tests with `MachineApplicable` lints.

@ghost

Rollup of 10 pull requests Successful merges: - #58717 (Add FromStr impl for NonZero types) - #59091 (Combine input and eval_always query types) - #59216 (Type dependent defs wrappers) - #59318 (rustc: Update linker flavor inference from filename) - #59320 (rustc: Allow using `clang` for wasm32 targets) - #59363 (#59361 Moved rustc edition opt to short list) - #59371 (ffi: rename VaList::copy to VaList::with_copy) - #59398 (Add a way to track Rustfix UI test coverage) - #59408 (compiletest: make path normalization smarter) - #59429 (When moving out of a for loop head, suggest borrowing it in nll mode) Failed merges: r? @ghost

rust-highfive assigned nikomatsakis Mar 24, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 24, 2019

rust-highfive assigned oli-obk and unassigned nikomatsakis Mar 24, 2019

phansch force-pushed the rustfix_coverage branch from e2b4d1e to 1e18c0e Compare March 24, 2019 13:57

This comment has been minimized.

Sign in to view

phansch force-pushed the rustfix_coverage branch from 1e18c0e to 02b8533 Compare March 24, 2019 16:04

phansch mentioned this pull request Mar 24, 2019

Crash in parse_snippet rust-lang/rustfix#164

Closed

zackmdavis reviewed Mar 24, 2019

View reviewed changes

src/tools/compiletest/src/common.rs Outdated Show resolved Hide resolved

phansch added 2 commits March 25, 2019 06:57

Fix two bootstrap tests

d485ebf

s/lints/diagnostics

98d7c54

Not all suggestions come from lints

Save coverage file in build_base path, not /tmp

d808bd4

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 25, 2019

phansch mentioned this pull request Mar 26, 2019

Add support for rustfix coverage tracking Manishearth/compiletest-rs#167

Merged

Centril mentioned this pull request Mar 28, 2019

Rollup of 10 pull requests #59487

Merged

bors merged commit d808bd4 into rust-lang:master Mar 29, 2019

phansch deleted the rustfix_coverage branch April 1, 2019 05:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a way to track Rustfix UI test coverage #59398

Add a way to track Rustfix UI test coverage #59398

phansch commented Mar 24, 2019

rust-highfive commented Mar 24, 2019

phansch commented Mar 24, 2019

This comment has been minimized.

rust-highfive commented Mar 24, 2019

phansch commented Mar 25, 2019

oli-obk commented Mar 25, 2019

phansch commented Mar 25, 2019

oli-obk commented Mar 25, 2019

bors commented Mar 25, 2019

phansch commented Mar 26, 2019

oli-obk commented Mar 26, 2019

phansch commented Mar 26, 2019

Add a way to track Rustfix UI test coverage #59398

Add a way to track Rustfix UI test coverage #59398

Conversation

phansch commented Mar 24, 2019

rust-highfive commented Mar 24, 2019

phansch commented Mar 24, 2019

This comment has been minimized.

rust-highfive commented Mar 24, 2019

phansch commented Mar 25, 2019

oli-obk commented Mar 25, 2019

phansch commented Mar 25, 2019

oli-obk commented Mar 25, 2019

bors commented Mar 25, 2019

phansch commented Mar 26, 2019

oli-obk commented Mar 26, 2019

phansch commented Mar 26, 2019