Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn if multiple tests run when trying to run a single test by name #2832

Closed
escamoteur opened this issue Sep 29, 2020 · 8 comments
Closed
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is enhancement
Milestone

Comments

@escamoteur
Copy link

Have a look at this screencast
https://www.screencast.com/t/j0utedFnzL

We found out the reason is that there is another test with the same name further up the file

Versions (please complete the following information):

  • VS Code version: 1.49.1
  • Dart extension version: v3.14.1
  • Dart/Flutter SDK version:1.22.0
@DanTup
Copy link
Member

DanTup commented Sep 30, 2020

This is unfortunately a known limitation. I don't believe pub run test has any way to run a single instance of a test if it has the exact same name (including group) as another test.

I don't think it's very common to have this, and they should both show up in the test tree (though trying to re-run either will re-run both), so hopefully it's easy to spot and correct.

@DanTup DanTup added this to the Backlog milestone Sep 30, 2020
@DanTup DanTup added blocked on dart / flutter Requires a change in Dart or Flutter to progress in testing Relates to test execution of Dart/Flutter tests for end users labels Sep 30, 2020
@escamoteur
Copy link
Author

Don't you think it would make sense to give a warning in such a case? especially as tests are often copied something like this can easily happen and the result is conpletely confusing

@DanTup
Copy link
Member

DanTup commented Sep 30, 2020

Yep, I do - but unfortunately it's not very simple to do reliably, as tests can be dynamic (include variable names, etc.).

There's an open issue in package:test discussing some possible improvements in this area:

dart-lang/test#1328

And some related things here:

dart-lang/test#1332

Hopefully these will help improve some of these things here.

@DanTup
Copy link
Member

DanTup commented Dec 8, 2020

I've excluded when test names contain variables (since that's more tricky), but for the most common case we'll know show a warning at the end of a test run if we expected only a single test to run (that is, you're not running a group, there is exactly one test name, and it does not contain a $) but multiple did:

Screenshot 2020-12-08 at 11 38 23

@DanTup DanTup removed the blocked on dart / flutter Requires a change in Dart or Flutter to progress label Dec 8, 2020
@DanTup DanTup modified the milestones: Backlog, v3.18.0 Dec 8, 2020
@DanTup DanTup changed the title if more than one test has the same name clicking on the debug or run link always executes the one further up Warn if multiple tests run when trying to run a single test by name Dec 8, 2020
@escamoteur
Copy link
Author

I think that should be enough in most cased. will close this now.
what happens if you click run on one of the tests that have a brother with the same name?

@DanTup DanTup reopened this Dec 8, 2020
@DanTup
Copy link
Member

DanTup commented Dec 8, 2020

(Re-opening as I haven't landed the change yet, but it'll close when I do :-))

what happens if you click run on one of the tests that have a brother with the same name?

I'm not sure what you mean - isn't this the same case described above? Or do you mean a non-sibling test? That should work as expected, as the "test name" we construct when running from code lens is actually a full name of all of the groups prefixed to it. It's only if the complete name matches that this will trigger. This is similar to how if you have two tests named "foo" in different groups, running one of them doesn't run the other today already.

@escamoteur
Copy link
Author

ah, I reread your comment about. I wasn't sure if it were possible that just one but the wrong test is executed.

@DanTup
Copy link
Member

DanTup commented Dec 8, 2020

It shouldn't be - we should always run the test that you wanted, but if there is another test with the exact same name, that may run too. All we do now is count how many tests ran, and if it's > 1 but we had run in a way that we expected only one (eg. a single test name) we'll show the warning.

@DanTup DanTup closed this as completed in 46fb2ef Dec 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is enhancement
Projects
None yet
Development

No branches or pull requests

2 participants