The flutter test command should require a flutter_test or test_api dependency (and check for it).

[jakemac53]

Use case

The flutter test command itself is built with a certain version of the test package (whatever is pinned in the repo at the time the SDK was published). If a user is only using the regular test package, then there is nothing to ensure that the version is compatible with that of the flutter test command.

See #91059 as an example failure.

Proposal

The flutter test command should check for a dependency on flutter_test, and fail (or at least warn) if one does not exist. This package pins the test_api package to a compatible version, solving the issue.

[Hixie]

Would it make sense to test for the right version of test_api, rather than flutter_test? (In the message we should suggest depending on flutter_test, certainly, but I feel like if someone is doing the dance to point to the right test_api version that should work too; they don't have to use flutter_test with flutter test.)

[jakemac53]

I would probably just depend on flutter_test, as it will take into account all the possible deps and make sure they are compatible. Being an sdk dependency it is already downloaded and essentially free (assuming it's a dev dependency).

In practice it's likely that just enforcing test_api is compatible would fix most issues though.

[Hixie]

I guess asking for it to be in the pubspec.yaml as a dev dependency is probably ok. It doesn't require actually importing it.

[jakemac53]

Note dart-lang/test#1977, this is another good example of why we should be requiring this dependency even if people don't use the package as an import.

[natebosch]

This is also what webdev does - it checks for appropriate versions of packages like build_web_compilers.

If any tool is going to operate with packages from the pub solve, but not be installed with that same pub solve, it should validate any assumptions it makes about how it can operate with those packages.

From the user standpoint a dev_dependency on the SDK vendored version of flutter_test is a far superior UX to a pinned dependency on test_api. With the former they can be compatible across Flutter SDK releases.

[polina-c]

DevTools just faced the issue

[guyluz11]

Not sure if it was mentiond already but wanted to point out that in the documentation of flutter it is written that

Add the test or flutter_test dependency.

https://docs.flutter.dev/cookbook/testing/unit/introduction

[jakemac53]

That documentation should also be updated as well to just always tell you to add the flutter_test dependency I suppose?

It feels a bit weird, but it does make sense, based on how the flutter test command ships. If you run dart test without a test dependency it will make you add that dependency, FWIW. So it also makes sense for flutter test to require a flutter_test dependency, even if it technically can work without it, it isn't safe to run it without it.

[devoncarew]

@christopherfujino - is this something we could address in the near term? We see issues related to this when we make changes to package:test_api (and related). We have some near term refactoring we'd like to do to help separate out package:test and package:matcher (and eventually allow us to support the newer package:checks).

I believe the desired resolution here is still the one from the first comment? (The flutter test command should check for a dependency on flutter_test, and fail (or at least warn) if one does not exist.)

[christopherfujino]

@christopherfujino - is this something we could address in the near term? We see issues related to this when we make changes to package:test_api (and related). We have some near term refactoring we'd like to do to help separate out package:test and package:matcher (and eventually allow us to support the newer package:checks).

I believe the desired resolution here is still the one from the first comment? (The flutter test command should check for a dependency on flutter_test, and fail (or at least warn) if one does not exist.)

SGTM

[flutter-triage-bot]

This issue is assigned to @christopherfujino but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

[jakemac53]

@christopherfujino this is about to come up again when we change the wire format for the Runtime class here dart-lang/test#2144. This type of change should not be breaking but it ends up breaking because there is no forced constraint on package:test_api when using flutter test, so the wire format between the tool and the package don't line up.

I think we can probably make it backwards compatible, but it would be great if we didn't have to, otherwise we are stuck with this tech debt until we decide to knowingly break users on some older version of flutter.

[christopherfujino]

@christopherfujino this is about to come up again when we change the wire format for the Runtime class here dart-lang/test#2144. This type of change should not be breaking but it ends up breaking because there is no forced constraint on package:test_api when using flutter test, so the wire format between the tool and the package don't line up.

I think we can probably make it backwards compatible, but it would be great if we didn't have to, otherwise we are stuck with this tech debt until we decide to knowingly break users on some older version of flutter.

@jakemac53 thanks for the heads up! Assigning @eliasyishak to fix.

[christopherfujino]

@eliasyishak is this still on your radar?

[eliasyishak]

Still tracking on my todos but haven't got around to it yet as I figure out how to handle the breaking change that comes with it