Evaluate code in Debug Console during integration tests

[bartekpacia]

Describe the bug

TL;DR I can't evaluate code in Debug Console when running integration tests. I don't know if I'm doing something wrong or if this is just not possible.

Also, I can't find a way to run (using VS Code's launch.json) normal widget tests with flutter run test/widget_test.dart (=running on device) instead of flutter test test/widget_test.dart (running in isolation). This might be another issue, though.

To Reproduce

TL;DR Run any Flutter integration test with VS Code and try to evaluate Dart code in the Debug Console.

Create .vscode/launch.json in the root of a Flutter project with the following contents:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "flutter run app",
      "request": "launch",
      "type": "dart",
      "program": "lib/main.dart"
    },
    {
      "name": "flutter run integration tests",
      "request": "launch",
      "type": "dart",
      "program": "integration_test/example_test.dart"
    },
  ]
}

1. Clone the repository of the Patrol framework
2. Open the example app directory with VS Code ($ code .). It's located in packages/patrol/example
3. Now that you're in the example app's dir, put a breakpoint somewhere in the integration_test/example_test.dart file, for example on line 16:

   await $(FloatingActionButton).tap();

4. Run the flutter run integration tests configuration from launch.json
5. Try to evaluate some code in the Debug Console.
6. See _compileExpression: No compilation service available; cannot evaluate from source..

Expected behavior

Being able to evaluate Dart code in Debug Console when the test is stopped on a breakpoint.

This is how it works with widget tests running in isolation:

Running.widget.test.in.isolation.mov

System info

• Operating System and version: macOS 12.6 on MacBook Air M1
• VS Code version: 1.72.0
• Dart extension version: v3.50.0
• Dart/Flutter SDK version: Dart 2.18.2, Flutter 3.3.3
• Target device: Android emulator (Pixel 5 API 33)

[DanTup]

Run any Flutter integration test with VS Code and try to evaluate Dart code in the Debug Console.

This is unfortunately a known limitation. It requires work in Flutter to hook up an compilation service for expression evaluation. There's an issue tracking that at flutter/flutter#79439.

Also, I can't find a way to run (using VS Code's launch.json) normal widget tests with flutter run test/widget_test.dart (=running on device) instead of flutter test test/widget_test.dart (running in isolation). This might be another issue, though.

You can do this by adding "runTestsOnDevice": true to your launch.json to run normal tests on a device.

You can also set this up to add additional CodeLens links with something like this:

{
	"name": "Test on Device",
	"request": "launch",
	"type": "dart",
	"runTestsOnDevice": true,
	"codeLens": {
		"for": [
			"run-test-file",
			"debug-test-file"
		],
		"path": "test",
		"title": "${debugType} on Device"
	}
},

This will add additional "Run on Device" and "Debug on Device" links above the main function in your test files:

This is only possible for the whole file (not individual tests), because flutter run is not able to run individual tests/groups.

Hope this helps!

[bartekpacia]

Thanks for answering my second little question. Works, indeed :)

But it's sad that we can't evaluate Dart code in Debug Console when running integration tests. I hope this gets implemented soon.

Thanks for your help.

[DanTup]

Yeah, I understand. Unfortunately I don't understand how the compilation service works well enough to know whether it's a big change or not. I would encourage adding 👍 's to that Flutter issue if you haven't already.

I'll close this issue as there's actually nothing to do in the VS Code extension. Once implemented in Flutter, it will just start working here.

[bartekpacia]

@DanTup Now I got this idea: is it possible to run a test file in the integration test directory with flutter run, not flutter test?

I'm thinking about something like:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "run app",
      "request": "launch",
      "type": "dart",
      "program": "lib/main.dart"
    },
    {
      "name": "run integration test",
      "request": "launch",
      "type": "dart",
      "runTestsOnDevice": true,
      "executeWithFlutterRun": true, // <-- let's assume that this flag exists
      "program": "integration_test/example_test.dart"
    }
  ]
}

I haven't found a way to do the above.

In other words: when I run the run integration test configuration, it does flutter test integration_test/example_test.dart. I'd like to change it to run flutter run -t integration_test/example_test.dart instead.

This would have the following benefits:

• we have the compilation service, so we're able to execute arbitrary Dart code in the Debug Console
• we can hot restart the integration test, which allows for much faster iteration and better DX

Of course, this approach isn't ideal – we'll lose the test reporting ability (and probably more flutter test-specific functionality.

But as a temporary workaround, I think it's an approach worth researching.

[DanTup]

@bartekpacia that's essentially what runTestsOnDevice does, however it only currently applies to unit tests (because integration tests already run on a device). We could perhaps add an override here, although it feels kinda clunky and much worse than being fixed in Flutter.

Let me ping on that issue and see if anyone can give an idea of the scale of work and pointers to see if it might be something I could contribute there.