Skip to content

"Run All Tests" passes all individual suite names for all projects when only some projects have exclusions #4553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dmrickey opened this issue May 18, 2023 · 3 comments
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is bug
Milestone

Comments

@dmrickey
Copy link

Describe the bug
On windows, test directories with too many test files output is just The command line is too long // Exited (1)

To Reproduce

  1. Be on windows
  2. have a directory with lots of tests
  3. the tests won't be run

Expected behavior
Should pass a single folder and not each individual test file

Screenshots
image
I've got 34 lines here with roughly 125 characters each.

Please complete the following information:

  • Operating System and version: Win 11 Business 22621.1265
  • VS Code version: 1.78.2
  • Dart extension version: v3.64.0
  • Dart/Flutter SDK version: 2.19.6
  • Target device (if the issue relates to Flutter debugging): simply running tests on windows
@DanTup
Copy link
Member

DanTup commented May 18, 2023

This happens when we're not running "everything" (such as if there are some exclusions that prevent us just doing flutter run test test).

Unfortunately, our check for whether we're running everything is for the whole run, and not per-project:

const isRunningAll = !request.include?.length && !request.exclude?.length;

This means if you exclude a test from ProjectA, then ProjectB will also be run by including all of the filenames.

At a minimum, we should attempt to group the exclusions into their projects, so we can still run "all" for the projects without exclusions.

Additionally:

  • We should use relative paths if possible, because these are currently absolute and exceeding the command line limit faster earlier necessary
  • Consider splitting up large runs into multiple debug sessions if we know they'll exceed the limit

@DanTup DanTup added this to the v3.66.0 milestone May 18, 2023
@DanTup DanTup added the in testing Relates to test execution of Dart/Flutter tests for end users label May 18, 2023
@DanTup DanTup changed the title Running tests on windows is passing each individual test file instead of a single folder glob "Run All Tests" passes all individual suite names for all projects when only some projects have exclusions (plus uses long absolute paths) May 18, 2023
@DanTup
Copy link
Member

DanTup commented May 18, 2023

Note to self: ping @dmrickey on Discord when a fix is available as a pre-release extension.

@DanTup DanTup changed the title "Run All Tests" passes all individual suite names for all projects when only some projects have exclusions (plus uses long absolute paths) "Run All Tests" passes all individual suite names for all projects when only some projects have exclusions May 20, 2023
@DanTup DanTup modified the milestones: v3.66.0, v3.68.0 May 31, 2023
@DanTup DanTup closed this as completed in b4d501f Jun 19, 2023
@DanTup
Copy link
Member

DanTup commented Jun 19, 2023

@dmrickey I pinged you on Discord, but in case you miss it, this is available in the v3.67.20230619 pre-release I just pushed. Please let me know if it doesn't/doesn't resolve the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is bug
Projects
None yet
Development

No branches or pull requests

2 participants