fix(test_api): 0.5.0 breaks flutter test for projects that dont depend on flutter_test

[renancaraujo]

After the release of version 0.5.0 of the test_api package, one may face a weird error message when running flutter test on a project that doesn't have a dependency on flutter_test (which comes from flutters SDK):

  Failed to load "/path/to/test/something_test.dart": type 'Null' is not a subtype of type 'Object' in type cast
  package:test_api  RemoteListener.start.<fn>.<fn>

After adding this to the pubspec file, tests run nomally:

dependency_overrides:
  test_api: 0.4.18

Additional context:

Minimal reproducible sample: https://github.com/renancaraujo/test_api_issue

Flutter doctor:

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.7.8, on macOS 12.6 21G115 darwin-arm64, locale en-PT)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.1)
[✓] VS Code (version 1.67.0)
[✓] Connected device (2 available)
[✓] HTTP Host Availability

• No issues found!

flutter packages get output:

Running "flutter pub get" in something...
Resolving dependencies... (1.3s)
+ _fe_analyzer_shared 57.0.0
+ analyzer 5.9.0
+ args 2.4.0
+ async 2.11.0
+ boolean_selector 2.1.1
+ collection 1.17.1
+ convert 3.1.1
+ coverage 1.6.3
+ crypto 3.0.2
+ file 6.1.4
+ frontend_server_client 3.2.0
+ glob 2.1.1
+ http_multi_server 3.2.1
+ http_parser 4.0.2
+ io 1.0.4
+ js 0.6.7
+ logging 1.1.1
+ matcher 0.12.15
+ meta 1.9.1
+ mime 1.0.4
+ mocktail 0.3.0
+ node_preamble 2.0.2
+ package_config 2.1.0
+ path 1.8.3
+ pool 1.5.1
+ pub_semver 2.1.3
+ shelf 1.4.0
+ shelf_packages_handler 3.0.1
+ shelf_static 1.1.1
+ shelf_web_socket 1.0.3
+ source_map_stack_trace 2.1.1
+ source_maps 0.10.12
+ source_span 1.9.1
+ stack_trace 1.11.0
+ stream_channel 2.1.1
+ string_scanner 1.2.0
+ term_glyph 1.2.1
+ test 1.24.0
+ test_api 0.5.0
+ test_core 0.5.0
+ typed_data 1.3.1
+ very_good_analysis 3.1.0 (4.0.0+1 available)
+ vm_service 11.3.0
+ watcher 1.0.2
+ web_socket_channel 2.3.0
+ webkit_inspection_protocol 1.2.0
+ yaml 3.1.1
Changed 47 dependencies!

Workrounds 🧠🔨

1. Stop running flutter test on projects that don't depend on flutter_test. See: The flutter test command should require a flutter_test or test_api dependency (and check for it). flutter/flutter#91107

2. Pin test_api to 0.4.18:

dev_dependencies:
  test_api: 0.4.18

[jakemac53]

Can you provide a full repro example of your setup? Are you trying to run flutter tests or Dart tests from flutter?

[jakemac53]

Some additional info that would be useful:

• flutter --version output
• the versions of your test_api, test_core, and test packages (or just the output of flutter pub get).

[renancaraujo]

@jakemac53 added some context to the issue and a reproducible sample.

[jakemac53]

Can you run this test fine with dart test?

[renancaraujo]

Can you run this test fine with dart test?

Yes.

[jakemac53]

cc @natebosch I think probably something changed in the communication channel between the test runner and tests, and without the flutter_test dependency then the pinned version of test_api doesn't take effect, and ultimately flutter test is not compatible.

[jakemac53]

I have no idea how we want to deal with this type of situation though... it is essentially unsupported that flutter test works for non-flutter tests, but I think it is common in practice to do, and it usually works.

[jakemac53]

I think that probably the best thing we can recommend is that if you want to run flutter test you need to depend on flutter_test, even if you don't actually import it, because you are relying on the flutter test runner, which does require certain versions of the test_* packages.

[renancaraujo]

I agree, I thought of opening an issue to let people be advised of this problem if they face it in the near future.

[jakemac53]

Happy to leave this open for a bit so people can hopefully discover it and find the workaround.

It might be helpful to that end if you add a "Workarounds" section to the top comment, suggesting either pinning test_api (note I would suggest doing this via regular dev_dependencies and not dependency_overrides), adding a dependency on flutter_test, or using dart test instead.

In general the flutter_test dependency is preferable - it should be harmless and is more future proof.

[jakemac53]

We should also consider filing an issue on the flutter repo, for flutter test to warn if you don't have that dependency, even if you aren't running flutter tests.

[jakemac53]

haha, I already filed this a long time ago :( flutter/flutter#91107

[renancaraujo]

Added that workarounds section 👍🏾

[natebosch]

I think probably something changed in the communication channel between the test runner and tests,

Yes, specifically the new 'compiler' field is missing from the flutter test usage.

de40c1c#diff-37c08d44d79fe33f695f7173107a674384e7fae4490ce3b14cda343d69d99490R55

I agree that there isn't anything we can do about this from the test_api library side - the tool needs to have this knowledge and act on it.

[renancaraujo]

Thanks for the immediate replies

[jakemac53]

Marking this as needs-info just so that it automatically gets closed in a little while (I think a couple weeks to a month). It doesn't actually need info though, haha.

[DanTup]

it is essentially unsupported that flutter test works for non-flutter tests, but I think it is common in practice to do

If this is common and the error comes from package:test_api, could it check for the null and throw clearer message? ("Field x is missing from foo, perhaps you're missing a dependency on flutter_test/test_api or should use 'dart test'?")

[natebosch]

I published a new version of test_api which is backwards compatible for this field in the communication between tests and the runner. It looks like flutter test is able to run these tests again with the current published version.

We should still add a check in flutter test that the project has a dependency on flutter_test, and packages should still prefer to use dart test when they don't have that dependency, but for the current package combinations it should work. I'll close this issue now that most users won't need a workaround.

[alaatrablse]

Thanks, it was a tough problem

[eliasyishak]

I published a new version of test_api which is backwards compatible for this field in the communication between tests and the runner. It looks like flutter test is able to run these tests again with the current published version.

We should still add a check in flutter test that the project has a dependency on flutter_test, and packages should still prefer to use dart test when they don't have that dependency, but for the current package combinations it should work. I'll close this issue now that most users won't need a workaround.

Hey @natebosch, I'm trying to look into fixing this issue by checking for the flutter_test dependency, but with your fix to make it backwards compatible, i'm having a hard time reproducing this error. Do you know how I can setup a simple repro to debug my fix?

[natebosch]

i'm having a hard time reproducing this error.

I don't think we should need to see the original error to verify the fix - even when the package versions are aligned in the pub solve and the test would pass I think we should validate the state of pubspec.yaml.