Add `--pass $mode` to compiletest through `./x.py` #61755

Centril · 2019-06-11T23:44:22Z

Adds a flag --pass $mode to compiletest, which is exposed through ./x.py.

When --pass $mode is passed, {check,build,compile,run}-pass tests will be forced to run under the given $mode unless the directive // ignore-pass exists in the test file.

The modes are explained in #61778:

check has the same effect as cargo check
build or compile have the same effect as cargo build
run has the same effect as cargo run

On my machine, ./x.py -i test src/test/run-pass --stage 1 --pass check takes 38 seconds whereas it takes 2 min 7 seconds without --pass check.

cc #61712

r? @petrochenkov

src/bootstrap/flags.rs

src/test/ui/print_type_sizes/packed.rs

src/test/ui/consts/const-eval/promoted_errors.rs

src/test/ui/lint/lint-type-overflow2.rs

src/test/ui/rmeta-lib-pass.rs

petrochenkov · 2019-06-12T09:39:56Z

It's an open question what exactly "no codegen" option should be used for // check-pass and --force check.
In case of --emit metadata I'm not sure whether

metadata files hit the "file name is too long" cases (Skip codegen for one UI test with long file path #60648) when emitted.
UPDATE: Turns out // skip-codegen already used --emit metadata, so it doesn't have issues with long file paths.
we probably don't really need the metadata files emitted for the check mode (unlike cargo), but on the other hand it would be nice to specify check/build/run as doing exactly the same thing as cargo.
Need to measure how expensive the metadata emission is.

Centril · 2019-06-12T17:33:59Z

@petrochenkov Fixed the comments :)

petrochenkov · 2019-06-12T18:43:56Z

This should probably wait for #61778.
Otherwise I'll have to revert the compiletest part of this PR to rebase #61778.
(Or let's squash and land, then reverting will be easy.)

Centril · 2019-06-12T19:01:46Z

OK Let's wait for #61778 to land first.

Centril · 2019-06-12T20:57:19Z

Squashed the commits a bit.

compiletest: Introduce `// {check,build,run}-pass` pass modes Pass UI tests now have three modes ``` // check-pass // build-pass // run-pass ``` mirroring equivalent well-known `cargo` commands. `// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases). `// build-pass` will compile and link the test without running it. `// run-pass` will compile, link and run the test. Tests without a "pass" annotation are still considered "fail" tests. Most UI tests would probably want to switch to `check-pass`. Tests validating codegen would probably want to run the generated code as well and use `run-pass`. `build-pass` should probably be rare (linking tests?). rust-lang#61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way. Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test. Perhaps some secondary CI can verify this invariant, but that's not super urgent. `// compile-pass` still works and is equivalent to `build-pass`. Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command. It can be removed some time in the future in a separate PR. cc rust-lang#61712

compiletest: Introduce `// {check,build,run}-pass` pass modes Pass UI tests now have three modes ``` // check-pass // build-pass // run-pass ``` mirroring equivalent well-known `cargo` commands. `// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases). `// build-pass` will compile and link the test without running it. `// run-pass` will compile, link and run the test. Tests without a "pass" annotation are still considered "fail" tests. Most UI tests would probably want to switch to `check-pass`. Tests validating codegen would probably want to run the generated code as well and use `run-pass`. `build-pass` should probably be rare (linking tests?). #61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way. Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test. Perhaps some secondary CI can verify this invariant, but that's not super urgent. `// compile-pass` still works and is equivalent to `build-pass`. Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command. It can be removed some time in the future in a separate PR. cc #61712

@petrochenkov

…petrochenkov Add `--pass $mode` to compiletest through `./x.py` Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`. When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file. The modes are explained in rust-lang#61778: - `check` has the same effect as `cargo check` - `build` or `compile` have the same effect as `cargo build` - `run` has the same effect as `cargo run` On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`. cc rust-lang#61712 r? @petrochenkov

RalfJung · 2019-06-29T09:26:35Z

This does not, however, give me the option to "filter" by mode, right? Say I wanted to only run non-pass tests, or only run *-pass tests?

Centril · 2019-06-29T09:28:50Z

@RalfJung That would be #61719.

petrochenkov · 2019-06-29T10:03:27Z

@RalfJung
Filtering was my first idea too, but after some thinking about testing scenarios (why I run tests and what I want from it) --pass check appeared to be a better approach.
With pass modes available I personally expect to never use filtering.

Centril · 2019-06-29T10:07:43Z

I do agree --pass check is what you want for regular development; but from a language team perspective I find the ability to do e.g. --filter-mode pass --test-args async to be useful if I want to extract all the relevant pass tests for async await (that assumes good organization for the async/await test, which is probably a great assumption but it's serviceable as a blunt instrument). That's not something I'd do every day, but it's still useful on occasion.

@ghost

Rollup of 7 pull requests Successful merges: - #61199 (Revert "Set test flag when rustdoc is running with --test option" ) - #61755 (Add `--pass $mode` to compiletest through `./x.py`) - #61818 (Issue #60709 test) - #62023 (publish_toolstate: don't use 'new' from inside the loop) - #62104 (Inform the query system about properties of queries at compile time) - #62163 (Avoid mem::uninitialized() in std::sys::unix) - #62204 (doc(libcore) Fix CS) Failed merges: r? @ghost

tests: Require run-pass tests without annotations to run successfully again Fixes #62775 (regression from #61755)

@Centril

Move run-pass tests to ui This is the second attempt at doing #53994 (which was previously reverted in #54530). The issue with inability to run the test suite in a faster way (#54047) that motivated the revert was recently addressed by #61755. r? @Centril

rust-highfive assigned petrochenkov Jun 11, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 11, 2019

Mark-Simulacrum reviewed Jun 12, 2019

View reviewed changes

src/bootstrap/flags.rs Outdated Show resolved Hide resolved

src/bootstrap/flags.rs Outdated Show resolved Hide resolved

src/test/ui/print_type_sizes/packed.rs Outdated Show resolved Hide resolved

petrochenkov mentioned this pull request Jun 12, 2019

Add --filter-mode=run-pass,compile-pass to compiletest #61719

Closed

petrochenkov reviewed Jun 12, 2019

View reviewed changes

src/test/ui/consts/const-eval/promoted_errors.rs Outdated Show resolved Hide resolved

petrochenkov reviewed Jun 12, 2019

View reviewed changes

src/test/ui/lint/lint-type-overflow2.rs Outdated Show resolved Hide resolved

petrochenkov reviewed Jun 12, 2019

View reviewed changes

src/test/ui/rmeta-lib-pass.rs Outdated Show resolved Hide resolved

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 12, 2019

petrochenkov mentioned this pull request Jun 12, 2019

compiletest: Introduce // {check,build,run}-pass pass modes #61778

Merged

Centril added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 12, 2019

Centril changed the title ~~Add --force check to compiletest through ./x.py~~ Add --pass check to compiletest through ./x.py Jun 12, 2019

This comment has been minimized.

Sign in to view

Centril added S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 12, 2019

Centril force-pushed the compiletest-force-check branch from 22c4a9e to 57b6a20 Compare June 12, 2019 20:56

This comment has been minimized.

Sign in to view

Centril added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. labels Jun 24, 2019

Centril added 3 commits June 24, 2019 07:58

bootstrap: pass '--pass' on to compiletest.

a56a6d7

compiletest: support '--pass check' and '// ignore-pass'.

54337fa

add '// ignore-pass' where applicable.

e364eb4

Centril mentioned this pull request Jun 28, 2019

Rollup of 8 pull requests #62219

Closed

petrochenkov mentioned this pull request Jun 29, 2019

Don't store locals that have been moved from in generators #61922

Merged

Centril mentioned this pull request Jun 29, 2019

Rollup of 7 pull requests #62226

Merged

bors merged commit 93077f3 into rust-lang:master Jun 29, 2019

Centril deleted the compiletest-force-check branch June 29, 2019 15:32

petrochenkov mentioned this pull request Jun 29, 2019

More systematic compile/run markup for "pass" UI tests #61712

Closed

Centril mentioned this pull request Jul 4, 2019

Document --pass $mode rust-lang/rustc-dev-guide#381

Closed

petrochenkov mentioned this pull request Jul 19, 2019

tests: Require run-pass tests without annotations to run successfully again #62797

Merged

bors added a commit that referenced this pull request Jul 21, 2019

Auto merge of #62797 - petrochenkov:run-pass-hole, r=Mark-Simulacrum

529828e

tests: Require run-pass tests without annotations to run successfully again Fixes #62775 (regression from #61755)

bors added a commit that referenced this pull request Jul 21, 2019

Auto merge of #62797 - petrochenkov:run-pass-hole, r=Mark-Simulacrum

787acc6

tests: Require run-pass tests without annotations to run successfully again Fixes #62775 (regression from #61755)

bors added a commit that referenced this pull request Jul 21, 2019

Auto merge of #62797 - petrochenkov:run-pass-hole, r=Mark-Simulacrum

25035ca

tests: Require run-pass tests without annotations to run successfully again Fixes #62775 (regression from #61755)

bors added a commit that referenced this pull request Jul 22, 2019

Auto merge of #62797 - petrochenkov:run-pass-hole, r=Mark-Simulacrum

4bc1ce7

tests: Require run-pass tests without annotations to run successfully again Fixes #62775 (regression from #61755)

petrochenkov mentioned this pull request Jul 27, 2019

Move run-pass tests to ui #63029

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `--pass $mode` to compiletest through `./x.py` #61755

Add `--pass $mode` to compiletest through `./x.py` #61755

Centril commented Jun 11, 2019 •

edited

Loading

petrochenkov commented Jun 12, 2019 •

edited

Loading

Centril commented Jun 12, 2019

This comment has been minimized.

petrochenkov commented Jun 12, 2019

Centril commented Jun 12, 2019

Centril commented Jun 12, 2019

This comment has been minimized.

RalfJung commented Jun 29, 2019

Centril commented Jun 29, 2019

petrochenkov commented Jun 29, 2019

Centril commented Jun 29, 2019 •

edited

Loading

Add --pass $mode to compiletest through ./x.py #61755

Add --pass $mode to compiletest through ./x.py #61755

Conversation

Centril commented Jun 11, 2019 • edited Loading

petrochenkov commented Jun 12, 2019 • edited Loading

Centril commented Jun 12, 2019

This comment has been minimized.

petrochenkov commented Jun 12, 2019

Centril commented Jun 12, 2019

Centril commented Jun 12, 2019

This comment has been minimized.

RalfJung commented Jun 29, 2019

Centril commented Jun 29, 2019

petrochenkov commented Jun 29, 2019

Centril commented Jun 29, 2019 • edited Loading

Add `--pass $mode` to compiletest through `./x.py` #61755

Add `--pass $mode` to compiletest through `./x.py` #61755

Centril commented Jun 11, 2019 •

edited

Loading

petrochenkov commented Jun 12, 2019 •

edited

Loading

Centril commented Jun 29, 2019 •

edited

Loading