Would be nice to have --coverage in flutter_driver

[eseidelGoogle]

@cbracken says we have a lot of the plumbing already, but it's still probably several days work, mostly due to edge-cases/error handling/testing.

Not something we need now, but we'll eventually want it. One use-case that has been brought to us is understanding how much of our widget library has performance testing coverage.

[edwin-hernandez-globant]

What was its a solution?

[paul-hammant]

There's no solution - team's still considering it I think. Pulling FlutterDriver reliably into build (CI) toolchains is hard for teams, so lots have not done it at all.

[Starcarr]

@paul-hammant @eseidelGoogle Just want to add here that my team has a lot of Driver tests that we are now trying to replace with Widget tests because we don't have coverage reports for Driver tests – btw we are using Gitlab for CI.

Funnily enough that has been proven difficult due to this issue, because currently Flutter seems to be very opinionated between what you can do in Widget tests when it comes to HTTP.

[paul-hammant]

@Starcarr Elsewhere, I've some WebDriver test suites that are doing deep testing of components for web apps. I blog about that here - https://paulhammant.com/2017/02/01/ui-component-testing/. I can't easily record JS coverage in the browser, and slurp that back to the runner in order to gather stats for the whole test suite towards a "78.4%" declaration at the end of the build. I say this because FlutterDriver and WebDriver are in the same boat here.

My colleagues have also engineered a test suite - "the component tests" for our Flutter codebase and use FlutterDriver to drive that. It eve show mock routes as it charges through tests. We like that suite, as it is faster and more stable than the following "full stack" suite. We need retries for the latter, and not for the former. So that component suite: we have to estimate the coverage for it, which is sad.

[bismarabia]

Any news regarding this issue? will flutter driver test support coverage report any time soon?

[Hixie]

We're moving away from flutter_driver in favour of extending flutter_test to work on devices. I'm going to close this for now.

[tudor07]

@Hixie Should we stop investing in flutter_driver tests and start converting flutter_driver tests to flutter_test? Is there a more official announcement about deprecating flutter_driver? Thanks.

[paul-hammant]

There was a better statement on Reddit or Twitter - I'll dig it up.

[nioncode]

@paul-hammant did you find the post regarding flutter_test getting compatible to run on devices to maybe replace flutter_test?

[xiongyoudou]

We're moving away from flutter_driver in favour of extending flutter_test to work on devices. I'm going to close this for now.

hello，can you tell us when will the flutter test can work on devices?

[flukejones]

We're moving away from flutter_driver in favour of extending flutter_test to work on devices. I'm going to close this for now.

I'm in the process of getting CI done for an app nearing production, and have had a few issues with swipes. So I searched and found this.. It's rather disheartening to see a single comment such as this with no other information when you're trying to get something critical done.

[tudor07]

@flukejones totally agree, proper testing tools are mandatory for production-ready apps... It's been 1 year and no updates. I wish Google would first make sure we have the tools to make quality mobile apps before jumping into supporting web

[Vayth]

We're moving away from flutter_driver in favour of extending flutter_test to work on devices. I'm going to close this for now.

Will the progress be tracked here, or is there another issue for tracking this?
Should I use driver for new mobile project?

[xiongyoudou]

We're moving away from flutter_driver in favour of extending flutter_test to work on devices. I'm going to close this for now.

so ,what's the progress now please?I am going to develope the flutter CI,I am confused!

[jamesdixon]

@Hixie any chance you can provide an update so we know what direction to head? Had no idea that flutter_driver was being deprecated!

[flukejones]

This is quite probably the most frustrating thing I've encountered... From the flutter docs:

Unit tests and widget tests are handy for testing individual classes, functions, or widgets. However, they generally don’t test how individual pieces work together as a whole, or capture the performance of an application running on a real device. These tasks are performed with integration tests.

So the comment that flutter_driver is being depreciated in favour of the framework that is used for unit tests is out of order and frustrating as heck to encounter when you are busting your guts to ensure the integrity of a production app, especially when said comment has no followups, or hints as to how to use flutter_test in place of flutter_driver.

Can somebody please sort this mess out...

[KhaleelShaheenDC]

@eseidelGoogle 👆

[tudor07]

@eseidel @eseidelGoogle please?

[paul-hammant]

Maybe FlutterDriver could be moved to own repo on GitHub. The community might find it easier to make pull requests for it, and push it ahead of Google's own wishes for the same.

Not least would be implementation of the https://www.w3.org/TR/webdriver/ spec. Java's Swing technology has a UI-clicking test runner called Marathon, and the team that maintains that added compatibility with that spec - https://marathontesting.com/javadriver/. Here's me with some very fast Marathon & WebDriver tests for a Swing MVC demo project - https://github.com/paul-hammant/swing_component_testing

[muzuro]

We're moving away from flutter_driver in favour of extending flutter_test to work on devices. I'm going to close this for now.

We need a way to test our mobile app with real backend using http requests. I wonder what is offical flutter position should flutter driver be used or widget test with HttpOverrides.global be used for e2e testing.

[neiljaywarner]

@Hixie @eseidelGoogle @timsneath any update please?

[Hixie]

Closed bugs are unlikely to get updates. Please file a new bug if there is still something that needs to happen.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.