Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run tests using line/col to support groups/test with dynamic names #4021

Closed
5 tasks done
MiniSuperDev opened this issue Jun 15, 2022 · 12 comments · Fixed by #4487
Closed
5 tasks done

Run tests using line/col to support groups/test with dynamic names #4021

MiniSuperDev opened this issue Jun 15, 2022 · 12 comments · Fixed by #4487
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is enhancement
Milestone

Comments

@MiniSuperDev
Copy link

MiniSuperDev commented Jun 15, 2022

Added by @DanTup:

Some test cases to ensure are handled:


Describe the bug
I have some functions to give a specific format to the descriptions of the groups and tests, like indentation, in this way it is easier to read them in the console, but when I execute a specific test, giving run in that group or test it tells me:

No tests match regular expression "^groupDescriptionFormatted\('group 1'\)".

To Reproduce
this is a minimal example:

import 'package:test/test.dart';

String groupDescriptionFormatted(String description) {
  return description;
}

String subGroupDescriptionFormatted(String description) {
  return '\n    $description';
}

void main() {
  group(
    groupDescriptionFormatted('group 1'),
    () {
      group(subGroupDescriptionFormatted('sub group 1'), () {
        test('test', () {
          expect(true, isTrue);
        });
      });
    },
  );
  group(
    groupDescriptionFormatted('group 2'),
    () {
      group(subGroupDescriptionFormatted('sub group 1'), () {
        test('test', () {
          expect(true, isTrue);
        });
      });
    },
  );
}

If you press run or debug on a specific group or test, you can see the error, an also in the testing tab you can see that the names are the function instead of the real test name.
image

If you press run or debug on main, or use the command run all tests, it runs normal but also in the testing tab keep the names are the function instead of the real test name, and now also the real tests names.

image

Thank you.

Please complete the following information:
Operating System and version: Windows 10
VS Code version: 1.68.0
Dart extension version: 3.42.0
Dart/Flutter SDK version: flutter 3.0.1

@DanTup
Copy link
Member

DanTup commented Jun 20, 2022

Currently there are some limitations because of how tests are run individually (by name). Without executing the tests, it's impossible for the extension to know what groupDescriptionFormatted('group 1') returns, so it's not possible to build a dart test` command that has the correct test name to run.

However, once you've run the test file, you should be able to run the tests from the explorer, or from the test icons in the gutter, because the previous runs names would have been captured.

package:test does now have some other ways we can run tests (such as by line number/column) which may allow us to handle this better, but it'll need some work to move over (and ensure we don't try to do it for older versions of package:test).

I'll keep this issue open to track that, as it looks like there isn't an existing issue tracking it right now.

@DanTup DanTup added this to the On Deck milestone Jun 20, 2022
@DanTup DanTup added is enhancement in testing Relates to test execution of Dart/Flutter tests for end users and removed is bug labels Jun 20, 2022
@DanTup DanTup changed the title When run an specific group or test where the description is got from a function, It Fails and get "No tests match regular expression" Run tests using line/col to support groups/test with dynamic names Jun 20, 2022
@jtdLab
Copy link

jtdLab commented Oct 2, 2022

Will running single tests/groups with dynamic descriptions via vs code be possible with this?

@DanTup
Copy link
Member

DanTup commented Oct 3, 2022

@jtdLab I'm hoping so, but I haven't done much testing yet (and it may depend on the exact configuration of the test). If you have some specific tests in mind, you can try running them from the terminal using the line/col like:

dart test "path/to/test.dart?line=10&col=2"

Where line/col are where your test definitions are. If that runs them, then it should work here. If it doesn't work, please post an example and I can do some digging.

@jtdLab
Copy link

jtdLab commented Oct 3, 2022

@DanTup im thinking about a scenario like this

foo_test.dart

// I want to run this test via the vs code Run | Debug (even though it has no description)
rendersText(MyWidget(), text: 'Foo');

my_shared_testing_library.dart

@isTest
void rendersText(Widget widget, {required String text}) {
  testWidgets('renders $text', (tester) async {
    // Some testing here
  });
}

Is this possible via line/col and not description?

@DanTup
Copy link
Member

DanTup commented Oct 3, 2022

@jtdLab I would hope/expect that to work. You can test it by running dart test foo_test.dart?line=1 (where 1 is the line it's on). I'm not sure how well we handle tests without names in the Test Runner tree though (when they are discovered from source).. we might need to fabricate a name (like "test on line 1") so there is some label for the tree node.

@jtdLab
Copy link

jtdLab commented Oct 3, 2022

@DanTup When running via the clickable Run | Debug above the test on v3.50.0 i get No tests match regular expression "^const MyWidget\(\)( \(variant: .*\))?$".

@DanTup
Copy link
Member

DanTup commented Oct 3, 2022

@jtdLab that's expected right now, using line/col has not been implemented in the extension yet.

You can use dart test foo_test.dart?line=1 from the terminal to verify where package:tests's support for line/cols works for this example. If it does, then once I update the extension to use this (implementing this issue), then it should also work from the Run/Debug CodeLens links.

@jtdLab
Copy link

jtdLab commented Oct 3, 2022

@DanTup Perfect thank you a lot

@jtdLab
Copy link

jtdLab commented Oct 3, 2022

@DanTup Let <LINE> be the line of the test. When running dart test foo_test.dart?line=<LINE> i get no matches found: foo_test.dart?line=<LINE> on test: ^1.21.6 so looks like this case is not covered there

@DanTup
Copy link
Member

DanTup commented Oct 4, 2022

I tested this using latest Flutter master (I missed you were using Flutter, so it should've been flutter test) and it seems to work as you'd like:

Screenshot 2022-10-04 at 12 31 16

So at least for a future version of Flutter, once this is implemented I believe what you're after works. (I tested it with rendersText in another file too, same result - I just inlined it here for the screenshot).

@DanTup
Copy link
Member

DanTup commented Nov 9, 2022

Started looking at this, but think we may be blocked on:

How to proceed will depend a little on the response to the test issue. If this is WAI and can't be changed, we may have to change how things work here a little more.

@DanTup DanTup modified the milestones: v3.54.0, On Deck Nov 10, 2022
@DanTup DanTup modified the milestones: On Deck, v3.62.0 Feb 27, 2023
@DanTup DanTup modified the milestones: v3.62.0, v3.64.0 Mar 23, 2023
@DanTup
Copy link
Member

DanTup commented Apr 19, 2023

In the next release (and the pre-release version just pushed today - v3.63.20240419), you can choose to have tests executed using their line numbers rather than their names. This fixes a number of issues including this one, caused by not always being able to statically compute the exact name of a test.

Test Invocation settings

Please let me know if you find any issues with this functionality!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is enhancement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants