Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upMove `compile-fail` tests to `ui` #44844
Comments
estebank
added
A-diagnostics
E-mentor
WG-compiler-errors
E-easy
labels
Sep 25, 2017
This comment has been minimized.
This comment has been minimized.
|
Has the relative performance of compile-fail vs. UI tests been evaluated? I worry that we'd be hurting our merge times if there's too much of a regression. I doubt it's anything serious, though, so no need to block this. |
This comment has been minimized.
This comment has been minimized.
|
Why are we doing this when compile-fail are easier to use, more readable and maintainable? |
This comment has been minimized.
This comment has been minimized.
I don't think anyone has (including me), but my gut feeling is that the difference isn't big enough to make a dent. I'll try to keep track of it though, as well as investigate any performance enhancements we might be able to make to the test suite itself.
I could debate all three points:
Beyond those three points, we want to add support for the current inline tests to There is another underlying reason, witch I haven't stated so far: I want to be able to audit the compiler output in it's totality in order to find common places for improvement, to write down unifying guidelines and applying them to any inconsistent output. Having only |
This comment has been minimized.
This comment has been minimized.
|
I tried to convert everything once and encountered two issues to solve
Most of tests still can be converted though. I think it's critical to implement and enable checking of |
This comment has been minimized.
This comment has been minimized.
|
It seems like a really strange idea... |
This comment has been minimized.
This comment has been minimized.
I'm counting on that.
But it should be easier for reviewers to catch them when they occur, IMO.
How's so? |
TimNN
added
the
C-tracking-issue
label
Sep 27, 2017
This comment has been minimized.
This comment has been minimized.
|
The goal of the compile-fail tests is different than the ui tests'. The first one is to ensure the expected error(s) come(s) out, even if the output changes whereas the second is to watch over the compiler output. That's why it seems strange to me that we're trying to merge the both of them. |
This comment has been minimized.
This comment has been minimized.
It's very difficult to write new compile-fail style tests testing more than one error without support of |
This comment has been minimized.
This comment has been minimized.
|
Which seems very complicated in case there are multiple lines for one error (help, notes, suggestions...). |
Manishearth
added
hacktoberfest
and removed
hacktoberfest
labels
Sep 28, 2017
This comment has been minimized.
This comment has been minimized.
|
Do we have a way to regenerate these tests on Windows? Also we need to document how to regenerate the tests somewhere discoverable, perhaps in the output of tests themselves? I'd also prefer just the diff of output be printed. These tests seem quite noisy. |
carols10cents
added
the
hacktoberfest
label
Sep 29, 2017
This comment has been minimized.
This comment has been minimized.
|
cc @pnkfelix @nikomatsakis (I precise again I'm against this change.) |
pnkfelix
added
the
T-compiler
label
Sep 30, 2017
This comment has been minimized.
This comment has been minimized.
|
I think the compiler team should discuss: 1. whether we want to go in this direction, and 2. if we do want to go in this direction, what prerequisites need to be met (e.g., consider @petrochenkov 's comment that we should implement and enable checking of |
pnkfelix
added
the
I-nominated
label
Sep 30, 2017
This comment has been minimized.
This comment has been minimized.
|
My opinion:
(I wish we had a lighterweight way to run the test runner, incidentally, without e.g. invoking it from |
This comment has been minimized.
This comment has been minimized.
|
@nikomatsakis Where is the test runner for Edit: Found the ui test runner and the compile-fail test runner. I'm going to do my best to understand what they're doing. |
This comment has been minimized.
This comment has been minimized.
|
Alright, I've upgraded the test runner for
I'm going through and fixing each one right now. This might take some time. |
This comment has been minimized.
This comment has been minimized.
|
Also, there doesn't appear to be any way to check "did you mean" stuff using commands... it's categorized by the json as kind: None, but there's no command //~ NONE. I'm just leaving them in as comments; they're still checked by stderr formatting diffs. |
This comment has been minimized.
This comment has been minimized.
|
There's 78 ui tests to fix, and a few features and special cases need to be added to my ui test runner code. Here's my work-in-progress stuff: https://github.com/Phlosioneer/rust Should I set up a pull request and just let it sit there while I work on this, for feedback? Or should I wait until I'm actually ready to submit the code? |
This comment has been minimized.
This comment has been minimized.
You are awesome. Thank you.
I had expected us not to parse json anymore but instead to go back to scraping from the output (which we used to do before we parsed JSON). Are you compiling everything twice? (I guess I can check your branch.) |
This comment has been minimized.
This comment has been minimized.
|
Opening up a |
This comment has been minimized.
This comment has been minimized.
|
@rfcbot fcp merge I'd like to propose transition compile-fail (and eventually run-pass) to UI tests, subject to the conditions that I described here. In particular, subject to the conditions that I've made my "positive" case before, in [this internals thread]. It is very similar to what @estebank had to say. @rust-lang/compiler, feel free to tell me why this is such a terrible idea. =) |
bors
added a commit
that referenced
this issue
Jul 16, 2018
bors
added a commit
that referenced
this issue
Jul 16, 2018
bors
added a commit
that referenced
this issue
Jul 16, 2018
bors
added a commit
that referenced
this issue
Jul 17, 2018
This comment has been minimized.
This comment has been minimized.
|
I would like to re-raise this issue in importance. We have great tooling for comparing the AST and MIR borrow checkers, but it is ALL based on |
nikomatsakis
added
the
WG-compiler-nll
label
Aug 7, 2018
nikomatsakis
added this to the Rust 2018 RC milestone
Aug 7, 2018
This comment has been minimized.
This comment has been minimized.
|
As such, I am marking this as an NLL Milestone issue. |
This comment has been minimized.
This comment has been minimized.
|
I'm currently taking a stab at this, will self-assign. |
davidtwco
self-assigned this
Aug 8, 2018
This comment has been minimized.
This comment has been minimized.
|
@davidtwco I'd advice doing it piece meal, not more than a couple hundred at a time, for your sanity's sake. You'll find that there are some tests with platform dependent output, which will require using some of the more esoteric features of ui tests (literal replacement), but there're tons of low hanging fruit that only needs to be moved with no changes. Also, the more files you involve the harder it'll be for you to merge it back as things change in master. |
bors
added a commit
that referenced
this issue
Aug 9, 2018
bors
added a commit
that referenced
this issue
Aug 10, 2018
bors
added a commit
that referenced
this issue
Aug 10, 2018
bors
added a commit
that referenced
this issue
Aug 10, 2018
bors
added a commit
that referenced
this issue
Aug 10, 2018
bors
added a commit
that referenced
this issue
Aug 11, 2018
bors
added a commit
that referenced
this issue
Aug 12, 2018
bors
added a commit
that referenced
this issue
Aug 13, 2018
bors
added a commit
that referenced
this issue
Aug 13, 2018
bors
added a commit
that referenced
this issue
Aug 13, 2018
bors
added a commit
that referenced
this issue
Aug 14, 2018
bors
added a commit
that referenced
this issue
Aug 14, 2018
This comment has been minimized.
This comment has been minimized.
|
This is completed as of #53196. As there are still some tests in |
estebank commentedSep 25, 2017
•
edited by kennytm
There are some cases where
compile-failerrors are still needed, asuierrors require the output to be exact and consistent across executions and platforms. That being said, the vast majority of errors can be safely moved.The process is straightforward:
git mv src/test/compile-fail/$FILE.rs src/test/ui/$FEATURE-DIR/$FILE.rs./x.py test --stage 1 src/test/ui./src/test/ui/update-all-references ./build/x86_64-apple-darwin/test/ui(replacex86_64-apple-darwinwith whatever environment you're working on)git add src/test/ui/$FEATURE-DIR/$FILE.stderrgit commit -m "Moving $FILE test to ui"You don't need to move only one file at a time, but avoid trying to do all at once, as that will probably cause too many conflicts with PRs being merged precluding these changes from landing.
The following is a list of all files and directories (without checking wether they can be moved at this time, I will audit this list at a later time):
Click to expand