Concurrency didn't work in single file with multiple tests

[EArminjon]

Description :
In the following example, I want to run tests in parallel to improve speed.
This tests took 1 minutes and 41 seconds...

Actual behavior :
If you run the code bellow, you will see that test are run one by one, group by group...

Expected behavior :
Test must be run in parallel

Code sample

import 'package:test/test.dart';

void main() {
  final List<int> items = List<int>.generate(10, (int index) => index);

  // Both groups took the same amount of time which is normal but It's too long
  group("For loop", () {
    for (final int item in items) {
      test(item.toString(), () async {
        await Future<void>.delayed(
          Duration(seconds: item),
        );
        print(item);
      });
    }
  });

  // Both groups took the same amount of time which is normal but It's too long
  group("One by one", () {
    test("1", () async {
      await Future<void>.delayed(
        const Duration(seconds: 1),
      );
      print("1");
    });

    test("2", () async {
      await Future<void>.delayed(
        const Duration(seconds: 2),
      );
      print("2");
    });

    test("3", () async {
      await Future<void>.delayed(
        const Duration(seconds: 3),
      );
      print("3");
    });

    test("4", () async {
      await Future<void>.delayed(
        const Duration(seconds: 4),
      );
      print("4");
    });

    test("5", () async {
      await Future<void>.delayed(
        const Duration(seconds: 5),
      );
      print("5");
    });

    test("6", () async {
      await Future<void>.delayed(
        const Duration(seconds: 6),
      );
      print("6");
    });

    test("7", () async {
      await Future<void>.delayed(
        const Duration(seconds: 7),
      );
      print("7");
    });

    test("8", () async {
      await Future<void>.delayed(
        const Duration(seconds: 8),
      );
      print("8");
    });

    test("9", () async {
      await Future<void>.delayed(
        const Duration(seconds: 9),
      );
      print("9");
    });

    test("10", () async {
      await Future<void>.delayed(
        const Duration(seconds: 10),
      );
      print("10");
    });
  });
}

Console log

➜  fvm dart test test/tmp_test.dart --concurrency=10
00:00 +0: For loop 0                                                                                                                                                                                   
0
00:01 +1: For loop 1                                                                                                                                                                                   
1
00:03 +2: For loop 2                                                                                                                                                                                   
2
00:06 +3: For loop 3                                                                                                                                                                                   
3
00:10 +4: For loop 4                                                                                                                                                                                   
4
00:15 +5: For loop 5                                                                                                                                                                                   
5
00:22 +6: For loop 6                                                                                                                                                                                   
6
00:29 +7: For loop 7                                                                                                                                                                                   
7
00:37 +8: For loop 8                                                                                                                                                                                   
8
00:46 +9: For loop 9                                                                                                                                                                                   
9
00:47 +10: One by one 1                                                                                                                                                                                
1
00:49 +11: One by one 2                                                                                                                                                                                
2
00:52 +12: One by one 3                                                                                                                                                                                
3
00:56 +13: One by one 4                                                                                                                                                                                
4
01:01 +14: One by one 5                                                                                                                                                                                
5
01:07 +15: One by one 6                                                                                                                                                                                
6
01:14 +16: One by one 7                                                                                                                                                                                
7
01:22 +17: One by one 8                                                                                                                                                                                
8
01:31 +18: One by one 9                                                                                                                                                                                
9
01:41 +19: One by one 10                                                                                                                                                                               
10
01:41 +20: All tests passed!   

Flutter doctor -v

[✓] Flutter (Channel stable, 3.13.7, on macOS 14.1.1 23B81 darwin-arm64, locale fr-FR)
    • Flutter version 3.13.7 on channel stable at /Users/earminjon/fvm/versions/3.13.7
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 2f708eb839 (3 months ago), 2023-10-09 09:58:08 -0500
    • Engine revision a794cf2681
    • Dart version 3.1.3
    • DevTools version 2.25.0

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/earminjon/Library/Android/Sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/earminjon/Library/Android/Sdk
    • Java binary at: /Users/earminjon/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A240d
    • CocoaPods version 1.14.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Users/earminjon/Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] IntelliJ IDEA Ultimate Edition (version 2023.3.2)
    • IntelliJ at /Users/earminjon/Applications/IntelliJ IDEA Ultimate.app
    • Flutter plugin version 77.0.1
    • Dart plugin version 233.13135.65

[✓] VS Code (version 1.76.2)
    • VS Code at /Users/earminjon/Downloads/Visual Studio Code.app/Contents
    • Flutter extension version 3.60.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-arm64   • macOS 14.1.1 23B81 darwin-arm64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 120.0.6099.216 [✓] Network resources
    • All expected network resources are available.

• No issues found!

[jakemac53]

Concurrency is only supported between separate test suites, so the way to get concurrency in this situation today would be to split up this suite into multiple suites.

The semantics of running individual tests within a suite concurrently are probably a lot harder to define than it seems - especially when you consider setUpAll/tearDownAll as well as any code that runs in main that isn't associated with a group/test.

It is also not always going to be the case that running tests within a suite concurrently would actually make them faster, sometimes it would be slower (if most of the work is actually in shared code that has to run for all tests, but only once).

[jakemac53]

If we did want to support this, one idea could be configuration via either dart_test.yaml or annotations on specific suites, to enable per-file concurrency for those particular suites.

[EArminjon]

Ty for your answer. I'm a bit lost. What is a test suite ? It's a file containing tests ?

[jakemac53]

It's a file containing tests ?

Correct, each test "suite" is a different program (which also makes running them concurrently trivial). They are typically _test.dart files, and they always have their own main function. Your entire file above is one "suite".

[EArminjon]

Okay understood. A solution could be to have a function like group named suite which will do that ?

[jakemac53]

A solution could be to have a function like group named suite which will do that ?

Possibly, but it would be quite a lot of engineering work to support, compared to the current model. On the other side the benefit is relatively minimal, compared to just creating separate files?

[EArminjon]

It will be painful to maintain for example if it's based on an enum or an Iterable logic. Also, when the list is 'huge' like 100, creating and maintain 100 files will be hard.

Edit : Also, on my app, the files which need this feature are not big at all, many time it's just few lines...

[jakemac53]

I would argue that it is easier to maintain 100 smaller files than one gigantic file personally. Although I understand some others have a preference for huge files. Either way the amount of code is essentially the same though.

If it were relatively trivial to implement, I think we would have support, since we generally want to support people working how they want to work. But in this case it is actually quite challenging.

Beyond just the actual implementation (which in itself could be quite complicated depending on the desired semantics), actually defining how it should work is also quite challenging.

As an example, note that any sort of concurrency at the suite level is also going to bring up a bunch of questions regarding variable state. Today it is common practice to create local or global variables, assign to them in setUp, and clean them up in tearDown. This is shared state between all tests though, so if they can run concurrently then you already have a major issue to resolve. I would suspect that this alone would break most tests. You could run every single test in its own isolate but this would result in most tests slowing down significantly, due to increased overhead.