Multiple tests execution

[grafovdenis]

Describe the bug
Once you're trying to run test, which description contains variable or type used in other test, multiple tests would be executed.
This could be annoying if you're using some kind of BLoC/Redux actions, which types may be located in multiple test descriptions.

Simular issue in /test repo: dart-lang/test#1760.
There we found out, that the plugin could interprete expressions like 'is $Object' incorrectly.

To Reproduce
Run 'is $Object' test.

import 'package:test/test.dart';

class MyObj {
  final number = 10;

  bool foo() => true;
}

void main() {
  group('$MyObj', () {
    late MyObj myObj;

    setUp(() {
      myObj = MyObj();
    });

    test('is $Object', () {
      expect(myObj, isA<Object>());
    });

    test('foo returns true and is an $Object', () {
      expect(myObj.foo(), isTrue);

      expect(myObj, isA<Object>());
    });

    test('number equals 10', () {
      expect(myObj.number, equals(10));
    });
  });
}

Expected behavior
Running 'is $Object' test, via IDE would execute only selected test.

Screenshots

It seems like plugin runs test like this.

Where $Object is an empty env. Which means, that we try to run test with 'is ' description, which also suits for the second test.

Please complete the following information:

• Operating System and version: macOS 12.3.1 (21E258)
• VS Code version: 1.71.0
• Dart extension version: v3.48.2
• Dart/Flutter SDK version: 2.17.3

[DanTup]

There are some known limitations in this area. When we run tests, we cannot use the literal name in the source if it contains variables:

% dart test test/dartcode4150_test.dart --plain-name "^\$MyObj is \$Object$"
No tests ran.                                                                                                                                                                                
No tests match "^$MyObj is $Object$".

We also cannot tell what $MyObj or $Object are to substitute their values, since they could be dynamic:

  final foo = DateTime.now().millisecond;
  group('$foo', () {

So right now, any variable is simple substitute with the regular expression pattern .* so it will match any value at runtime. This results in the behaviour that you see.

There are plans to improve this though, there were some changes made to package:test that will allow us to run tests by their line number. If we switch to this, we will be able to more accurately pinpoint a test to be run without relying on names that we can't compute the exact values for. That's being tracked in #4021, so I'll close this as a dupe of that. You can subscribe there for updates.

Thanks!

[DanTup]

Re-opening this for tracking to ensure this issue is fixed by the upcoming support for running tests by lines.

[DanTup]

In the next release (and the pre-release version just pushed today - v3.63.20240419), you can choose to have tests executed using their line numbers rather than their names. This fixes a number of issues including this one, caused by not always being able to statically compute the exact name of a test.

Please let me know if you find any issues with this functionality!