Support running/debugging Flutter Driver integration tests

[themightychris]

I'm working through this guide trying to get to a point where I can step through driver scripts and/or driven apps in VSCode: https://medium.com/flutter-community/hot-reload-for-flutter-integration-tests-e0478b63bd54

Running the terminal commands works, but I haven't been able to get either side to work when launched from VSCode yet. There seem to be two things in the way:

• Setting a known --observation-port via "args" in launch.json seems to get overridden by the extension
• There doesn't seem to be any way to run the driver program as a dart CLI command because the extension detects the project type from the workspace folder and offers no launch.json-level override. It gets launched on the Android device I just launched the app on, and I can breakpoint through it but it kills the app on the device to launch the driver

Here's my best guess so far:

    {
      "name": "Integration Tests: Launch App",
      "request": "launch",
      "type": "dart",
      "program": "test_driver/app.dart",
      "args": [
        "--dart-define=app.api=http://10.0.2.2:5378/api/v1", // special android IP for host, works
        "--observatory-port 4567", // doesn't seem to work
        "--disable-service-auth-codes"
      ],
      "flutterMode": "debug", // don't know if needed
    },
    {
      "name": "Integration Tests: Launch Driver",
      "request": "launch",
      "type": "dart",
      "program": "test_driver/app_test.dart",
      "deviceId": null, // this needs to run on the host computer, but it wants to run on the same device as the app
      "env": {
        "VM_SERVICE_URL": "http://127.0.0.1:4567"
      },
    },

[DanTup]

Could you try this version out for me?

• Download the.vsix file from https://github.com/Dart-Code/Dart-Code/releases/tag/v3.13.0-alpha.2
• Run the Extensions: Install from VSIX command from the VS Code command palette
• Browse to the file you downloaded
• Click the Restart button when prompted
• Remove your launch.json
• Open test_driver/app.dart and press F5 - this should now launch this file (the instrumented version)
• Wait for the app to launch
• Switch to test_driver.app_test.dart
• Press F5 to run the whole file, or use the CodeLens Run/Debug links to run individual tests

This should give you full debugging of both the app and the test, and you shouldn't need to mess around with launch configs. If you do want to be able to launch these from launch configs, you should be able to just strip out all the args/env you added:

 {
      "name": "Integration Tests: Launch App",
      "request": "launch",
      "type": "dart",
      "program": "test_driver/app.dart",
    },
    {
      "name": "Integration Tests: Launch Driver",
      "request": "launch",
      "type": "dart",
      "program": "test_driver/app_test.dart",
    },

Let me know if this seems to work ok for you (I've never used driver tests before, so I'm flying a little blind here). If you hit any issues, let me know (ether here, or once this issue is closed, in new issues). If it all works, you can remain on this test version and when the next stable release goes out you'll be automatically changed over to it (if you have auto updates for extensions enabled).

BTW - I made it automatically Hot Restart the app at the start of the test run, as it seemed weird to run with existing state.. if this feels weird, let me know (I did it because the sample tests here would fail if run multiple times due to the counter state not being reset).

[themightychris]

@DanTup it works!!! this is brilliant

[DanTup]

Great, thanks for confirming!

[themightychris]

@DanTup how long does it usually take for new versions to show up at https://marketplace.visualstudio.com/items?itemName=Dart-Code.dart-code

[DanTup]

@themightychris this feature isn't included in a release yet - it'll be in the version in the assigned milestone (v3.13.0) which is likely to ship around the end of the month (there was a release today, but it was a minor version to support using the Flutter SDK Snap). The issues are automatically closed when the work is merged to master.

For now, you should stick with the preview build from above - if you have auto-updates on, VS Code will automatically upgrade you to the stable v3.13.0 when it's published.

[tijanirf]

I'm working through this guide trying to get to a point where I can step through driver scripts and/or driven apps in VSCode: https://medium.com/flutter-community/hot-reload-for-flutter-integration-tests-e0478b63bd54

Running the terminal commands works, but I haven't been able to get either side to work when launched from VSCode yet. There seem to be two things in the way:

• Setting a known --observation-port via "args" in launch.json seems to get overridden by the extension
• There doesn't seem to be any way to run the driver program as a dart CLI command because the extension detects the project type from the workspace folder and offers no launch.json-level override. It gets launched on the Android device I just launched the app on, and I can breakpoint through it but it kills the app on the device to launch the driver

Here's my best guess so far:

    {
      "name": "Integration Tests: Launch App",
      "request": "launch",
      "type": "dart",
      "program": "test_driver/app.dart",
      "args": [
        "--dart-define=app.api=http://10.0.2.2:5378/api/v1", // special android IP for host, works
        "--observatory-port 4567", // doesn't seem to work
        "--disable-service-auth-codes"
      ],
      "flutterMode": "debug", // don't know if needed
    },
    {
      "name": "Integration Tests: Launch Driver",
      "request": "launch",
      "type": "dart",
      "program": "test_driver/app_test.dart",
      "deviceId": null, // this needs to run on the host computer, but it wants to run on the same device as the app
      "env": {
        "VM_SERVICE_URL": "http://127.0.0.1:4567"
      },
    },

Hi @themightychris, you mention here that you can use --dart-define inside args. I already tried but it's not working on the test driver, it is only working on my default app. Are there any gotchas that I need to carefully check?

[themightychris]

@tijanirf it's all super easy to do now, and built into the vscode extension. You don't need to pass --observatory-port or --disable-service-auth-codes in launch.json, you just need to launch the driver-wrapped version of your app from launch.json, and then when you open up your test suite in vscode the editor will be decorated with Run | Debug buttons at various scopes, just click them while you have the instrumented app running from vscode and it should all Just Work!

[tomasfly]

Any ideas how to debug integration tests using new integration_test package from SDK?

[DanTup]

@tomasfly it should generally just work since v3.22 (see https://dartcode.org/releases/v3-22/#flutter-integration-testing). If you're having issues with it not doing as you'd expect, please file a new issue with detailed steps and I'll take a look. Thanks!

[JaredEzz]

Does anyone have any tips for doing this in IntelliJ? Currently I just run flutter drive -d chrome with the target and driver args from the terminal which doesn't allow for debugging

[gabrielchaves7]

@DanTup Is it possible to debug integration test using chrome driver?. As @JaredEzz mentioned, right now I'm running on the terminal using flutter -d chrome drive --driver=test_driver/integration_test.dart --target=integration_test/main_test.dart --web-port=8080 but I would like to debug the tests and couldn't figure it out a way

[DanTup]

@gabrielchaves7 unfortunately I don't think this is currently possible. The ideal way would be for flutter test (which is now how we run non-web integration tests.. this issue is a little old and predates that) to support web, then things should just work from the editors. I thought there was an open issue in flutter/flutter about this, but I can't find one. It may be worth opening one (flutter test already supports all the flags needed for debugging, starting paused, parsing the test output to populate the test tree etc.).

I did wonder if in the meantime if you only need the debugging part (not test tree etc.) you could do this with a launch configuration that does an attach and a preLaunchTask to run flutter drive, but it seems like flutter drive doesn't support --start-paused, since running this:

 flutter drive --driver=test_driver/integration_test.dart --target=integration_test/counter_test.dart -d web-server --start-paused --host-vmservice-port=8822

...does not appear to start paused (as would be required for an attach configuration to be able to connect to it before the tests started running). Not sure if that's a bug or expected though.