Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way to track Rustfix UI test coverage #59398

Merged
merged 4 commits into from Mar 29, 2019

Conversation

Projects
None yet
6 participants
@phansch
Copy link
Contributor

commented Mar 24, 2019

This came out of the first Rustfix WG meeting.

One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with MachineApplicable suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing // run-rustfix headers.

This PR adds a --rustfix-coverage flag to ./x.py and compiletest to
list the files with the missing headers in /tmp/rustfix_missing_coverage.txt.

From that file we can create the tracking issue and at some point also
enforce the // run-rustfix flag on UI tests with MachineApplicable
lints.

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Mar 24, 2019

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

@phansch

This comment has been minimized.

Copy link
Contributor Author

commented Mar 24, 2019

r? @oli-obk

and cc @killercup I guess? =)

@phansch phansch force-pushed the phansch:rustfix_coverage branch from e2b4d1e to 1e18c0e Mar 24, 2019

@rust-highfive

This comment was marked as outdated.

Copy link
Collaborator

commented Mar 24, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0144ed58:start=1553435950773962568,finish=1553435951722892419,duration=948929851
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---

[00:04:09] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:04:09] tidy error: /checkout/src/bootstrap/flags.rs:195: line longer than 100 chars
[00:04:09] tidy error: /checkout/src/tools/compiletest/src/runtest.rs:2833: line longer than 100 chars
[00:04:09] tidy error: /checkout/src/tools/compiletest/src/main.rs:239: line longer than 100 chars
[00:04:11] some tidy checks failed
[00:04:11] 
[00:04:11] 
[00:04:11] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:04:11] 
[00:04:11] 
[00:04:11] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:11] Build completed unsuccessfully in 0:00:48
[00:04:11] Build completed unsuccessfully in 0:00:48
[00:04:11] Makefile:67: recipe for target 'tidy' failed
[00:04:11] make: *** [tidy] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:3149e89c
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Mar 24 14:03:34 UTC 2019
---
travis_time:end:19b0cfce:start=1553436215654064908,finish=1553436215659680859,duration=5615951
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:12154f5c
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:313f0182
travis_time:start:313f0182
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:2916f939
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Add a way to track Rustfix UI test coverage
This came out of the first Rustfix WG meeting.

One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with `MachineApplicable` suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing `// run-rustfix` headers.

This PR adds a `--rustfix-coverage` flag to `./x.py` and compiletest to
list the files with the missing headers in `/tmp/rustfix_missing_coverage.txt`.

From that file we can create the tracking issue and at some point also
enforce the `// run-rustfix` flag on UI tests with `MachineApplicable`
lints.

@phansch phansch force-pushed the phansch:rustfix_coverage branch from 1e18c0e to 02b8533 Mar 24, 2019

@phansch phansch referenced this pull request Mar 24, 2019

Closed

Crash in parse_snippet #164

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Mar 24, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0d355238:start=1553443553537255520,finish=1553443554484076371,duration=946820851
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:21:45] 
[01:21:45] running 9 tests
[01:21:45] iiiiiiiii
[01:21:45] 
[01:21:45]  finished in 0.158
[01:21:45] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:22:01] 
[01:22:01] running 120 tests
[01:22:27] .iiiii...i.....i..i...i..i.i...i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i.i 100/120
[01:22:32] .i......iii.i.....ii
[01:22:32] 
[01:22:32]  finished in 30.954
[01:22:32] travis_fold:end:test_debuginfo

---
[01:51:51]    Compiling ansi_term v0.11.0
[01:51:51]    Compiling difference v2.0.0
[01:51:52]    Compiling pretty_assertions v0.5.1
[01:51:52]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[01:51:56] error[E0063]: missing field `rustfix_coverage` in initializer of `flags::Subcommand`
[01:51:56]      |
[01:51:56] 1851 |         config.cmd = Subcommand::Test {
[01:51:56] 1851 |         config.cmd = Subcommand::Test {
[01:51:56]      |                      ^^^^^^^^^^^^^^^^ missing `rustfix_coverage`
[01:51:56] 
[01:51:56] error[E0063]: missing field `rustfix_coverage` in initializer of `flags::Subcommand`
[01:51:56]      |
[01:51:56] 1892 |         config.cmd = Subcommand::Test {
[01:51:56] 1892 |         config.cmd = Subcommand::Test {
[01:51:56]      |                      ^^^^^^^^^^^^^^^^ missing `rustfix_coverage`
[01:51:56] error: aborting due to 2 previous errors
[01:51:56] 
[01:51:56] For more information about this error, try `rustc --explain E0063`.
[01:51:56] error: Could not compile `bootstrap`.
---
[01:51:56] 
[01:51:56] 
[01:51:56] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:51:56] Build completed unsuccessfully in 0:41:59
[01:51:56] make: *** [check] Error 1
[01:51:56] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:2938cf98
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Mar 24 17:58:01 UTC 2019
---
43088 ./obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnutravis_time:end:13bc5fdf:start=1553450283476555580,finish=1553450283533642021,duration=57086441
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0925885e
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:10e0a26c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

phansch added some commits Mar 25, 2019

s/lints/diagnostics
Not all suggestions come from lints
@phansch

This comment has been minimized.

Copy link
Contributor Author

commented Mar 25, 2019

This will need a bump of the rustfix version as well, once rust-lang-nursery/rustfix#165 is merged and released.

@oli-obk

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2019

I don't think we should be writing to /tmp. Can you fetch the path that resolves to build in the default configuration (or is build hardcoded?) and dump the file in there?

@phansch

This comment has been minimized.

Copy link
Contributor Author

commented Mar 25, 2019

I don't think we should be writing to /tmp

Good point, fixed in d808bd4. It's now written to ./build/<host-triple>/test/ui/rustfix_missing_coverage.txt.

@oli-obk

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2019

@bors r+

now you've got me thinking about how I can get our CI to emit this metric on every build so I can create a graph

@bors

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2019

📌 Commit d808bd4 has been approved by oli-obk

@phansch

This comment has been minimized.

Copy link
Contributor Author

commented Mar 26, 2019

I think this won't succeed yet because it will still crash on some UI tests until rust-lang-nursery/rustfix#165 is merged =)

@oli-obk

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

Since we're not running the script in CI, is that problematic? I mean until it's merged and rustfix is updated in this repo, we can just remove the broken tests locally.

@phansch

This comment has been minimized.

Copy link
Contributor Author

commented Mar 26, 2019

Since we're not running the script in CI, is that problematic?

Oh, you're completely right! It will be fine then =)

phansch added a commit to phansch/compiletest-rs that referenced this pull request Mar 26, 2019

Add support for rustfix coverage tracking
Pretty much a copy of rust-lang/rust#59398 minus
the rustc bootstrap things.

This will allow rustfix coverage tracking in Clippy and other
libraries that emit `MachineApplicable` diagnostics.

Centril added a commit to Centril/rust that referenced this pull request Mar 28, 2019

Rollup merge of rust-lang#59398 - phansch:rustfix_coverage, r=oli-obk
Add a way to track Rustfix UI test coverage

This came out of the first Rustfix WG meeting.

One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with `MachineApplicable` suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing `// run-rustfix` headers.

This PR adds a `--rustfix-coverage` flag to `./x.py` and compiletest to
list the files with the missing headers in `/tmp/rustfix_missing_coverage.txt`.

From that file we can create the tracking issue and at some point also
enforce the `// run-rustfix` flag on UI tests with `MachineApplicable`
lints.

bors added a commit that referenced this pull request Mar 28, 2019

Auto merge of #59487 - Centril:rollup, r=Centril
Rollup of 10 pull requests

Successful merges:

 - #58717 (Add FromStr impl for NonZero types)
 - #59091 (Combine input and eval_always query types)
 - #59216 (Type dependent defs wrappers)
 - #59318 (rustc: Update linker flavor inference from filename)
 - #59320 (rustc: Allow using `clang` for wasm32 targets)
 - #59363 (#59361 Moved rustc edition opt to short list)
 - #59371 (ffi: rename VaList::copy to VaList::with_copy)
 - #59398 (Add a way to track Rustfix UI test coverage)
 - #59408 (compiletest: make path normalization smarter)
 - #59429 (When moving out of a for loop head, suggest borrowing it in nll mode)

Failed merges:

r? @ghost

bors added a commit that referenced this pull request Mar 28, 2019

Auto merge of #59487 - Centril:rollup, r=Centril
Rollup of 10 pull requests

Successful merges:

 - #58717 (Add FromStr impl for NonZero types)
 - #59091 (Combine input and eval_always query types)
 - #59216 (Type dependent defs wrappers)
 - #59318 (rustc: Update linker flavor inference from filename)
 - #59320 (rustc: Allow using `clang` for wasm32 targets)
 - #59363 (#59361 Moved rustc edition opt to short list)
 - #59371 (ffi: rename VaList::copy to VaList::with_copy)
 - #59398 (Add a way to track Rustfix UI test coverage)
 - #59408 (compiletest: make path normalization smarter)
 - #59429 (When moving out of a for loop head, suggest borrowing it in nll mode)

Failed merges:

r? @ghost

bors added a commit that referenced this pull request Mar 28, 2019

Auto merge of #59487 - Centril:rollup, r=Centril
Rollup of 10 pull requests

Successful merges:

 - #58717 (Add FromStr impl for NonZero types)
 - #59091 (Combine input and eval_always query types)
 - #59216 (Type dependent defs wrappers)
 - #59318 (rustc: Update linker flavor inference from filename)
 - #59320 (rustc: Allow using `clang` for wasm32 targets)
 - #59363 (#59361 Moved rustc edition opt to short list)
 - #59371 (ffi: rename VaList::copy to VaList::with_copy)
 - #59398 (Add a way to track Rustfix UI test coverage)
 - #59408 (compiletest: make path normalization smarter)
 - #59429 (When moving out of a for loop head, suggest borrowing it in nll mode)

Failed merges:

r? @ghost

@bors bors merged commit d808bd4 into rust-lang:master Mar 29, 2019

1 check passed

Travis CI - Pull Request Build Passed
Details

@phansch phansch deleted the phansch:rustfix_coverage branch Apr 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.