Terrible diagnostics when missing flutter_test dependency

[emshack]

Steps to Reproduce

I'm trying to follow the directions on flutter.io to run Flutter unit tests, but I'm having problems with the 'flutter test' command. If I copy over the following file exactly, and then try to test it with flutter test, it fails. Also, all of my dart tests which work using the 'pub run test' command fail on 'flutter test' with the same error. I tried upgrading flutter to the newest version, but still got the same error.

import 'package:test/test.dart';

void main() {
  test('this is a unit test', () {
    expect(42, 42);
  });
}

Flutter Doctor

[✓] Flutter (on Mac OS, channel master)
    • Flutter at /Users/emshack/flutter
    • Framework revision 596637ade4 (33 minutes ago), 2016-10-03 16:08:15
    • Engine revision 71f2871f00
    • Tools Dart version 1.20.0-dev.5.0

[✓] Android toolchain - develop for Android devices (Android SDK 23.0.2)
    • Android SDK at /Users/emshack/Library/Android/sdk
    • Platform android-23, build-tools 23.0.2
    • Java(TM) SE Runtime Environment (build 1.8.0_60-b27)

[✓] iOS toolchain - develop for iOS devices (Xcode 7.3.1)
    • XCode at /Applications/Xcode.app/Contents/Developer
    • Xcode 7.3.1, Build version 7D1014
    x ideviceinstaller not available; this is used to discover connected iOS devices.
      Install via 'brew install ideviceinstaller'.
    x ios-deploy not available; this is used to deploy to connected iOS devices.
      Install via 'brew install ios-deploy'.

[✓] Atom - a lightweight development environment for Flutter
    • flutter plugin version 0.2.5
    • dartlang plugin version 0.6.39

[✓] Connected devices
    • None

Logs and Crash Reports

flutter test lib/test/
00:00 +0 -1: loading [app directory]/lib/test/flutter_layout_test.dart 
  Failed to load "[app directory]/lib/test/flutter_test/flutter_layout_test.dart": Failed assertion: boolean expression must not be null
  package:test                                         test
  lib/test/flutter_test/flutter_layout_test.dart 4:3  main
  dart:async                                           _StreamController.add
  websocket_impl.dart 1111                             _WebSocketImpl._WebSocketImpl._fromSocket.<fn>
  dart:async                                           _EventSinkWrapper.add
  websocket_impl.dart 333                              _WebSocketProtocolTransformer._messageFrameEnd
  websocket_impl.dart 228                              _WebSocketProtocolTransformer.add

[abarth]

Interesting. What happens if you move the test directory to be a sibling of the lib directory and run flutter test without any arguments?

[Hixie]

Does flutter analyze give any interesting output?

[emshack]

@abarth
Doing this gives me the same error.

@Hixie
Nope, fixed the issues it surfaced and it's still giving the same error.

[eseidelGoogle]

I could be remembering wrong, but i thought the syntax for package:test was expect(42, equals(42))? https://github.com/dart-lang/test But I may also be mis-understanding the reported issue.

[emshack]

@eseidelGoogle
Using this syntax still gives me the same error.

[Hixie]

@eseidelGoogle it supports expect(42, 42) as well as a shorthand.

@emshack Can you attach the complete output of flutter test -v with the test being in [app directory]/test/foo_test.dart ?

[emshack]

emshack-macbookpro:[app directory] emshack$ flutter test -v
   0 ms • switching to directory Directory: 'test' to run tests
00:00 +0: loading [app directory]/test/foo_test.dart                                
 118 ms • running test package with arguments: [--, [app directory]/test/foo_test.dart]
00:00 +0 -1: loading [app directory]/test/foo_test.dart                             
  Failed to load "[app directory]/test/foo_test.dart": Failed assertion: boolean expression must not be null
  package:test              test
  foo_test.dart 4:3         main
  dart:async                _StreamController.add
  websocket_impl.dart 1111  _WebSocketImpl._WebSocketImpl._fromSocket.<fn>
  dart:async                _EventSinkWrapper.add
  websocket_impl.dart 333   _WebSocketProtocolTransformer._messageFrameEnd
  websocket_impl.dart 228   _WebSocketProtocolTransformer.add

00:00 +0 -1: Some tests failed.                                                                    
 432 ms • /Users/emshack/flutter/bin/cache/artifacts/engine/darwin-x64/sky_shell --disable-observatory --non-interactive --enable-checked-mode --packages=[app directory]/.packages /var/folders/c2/m74f07s13jbg8v0xgyf078dw00_60m/T/dart_test_listeneri4ipDx/listener.dart
   0 ms • test package returned with exit code 1
   0 ms • 'flutter test' took 605ms; exiting with code 1.
   0 ms • ensureAnalyticsSent: 75ms

[emshack]

(And the full contents of foo_test.dart):

import 'package:test/test.dart';

void main() {
  test('this is a unit test', () {
    expect(42, equals(42));
  });
}

[Hixie]

That's really really weird.

Can you try attaching the output of running flutter -v packages upgrade in [app directory]? I'm grabbing at straws here.

[emshack]

emshack-macbookpro:[app directory] emshack$ flutter -v packages upgrade
   1 ms • Running 'pub upgrade' in [app directory]...
4974 ms • [[app directory]] /Users/emshack/flutter/bin/cache/dart-sdk/bin/pub --verbosity=warning upgrade --no-packages-dir --no-precompile
   0 ms • 'flutter upgrade' took 5016ms; exiting with code 0.
   0 ms • ensureAnalyticsSent: 55ms

[emshack]

By the way, if I comment out the 'expect(42, equals(42));' line, I still get that error.

[abarth]

Ah! Got it.

clearzen:foo abarth$ flutter test
00:00 +0 -1: loading /Users/abarth/tmp/em/foo/test/foo_test.dart                                   
  Failed to load "/Users/abarth/tmp/em/foo/test/foo_test.dart": Failed assertion: boolean expression must not be null
  package:test              test
  foo_test.dart 4:3         main
  dart:async                _StreamController.add
  websocket_impl.dart 1111  _WebSocketImpl._WebSocketImpl._fromSocket.<fn>
  dart:async                _EventSinkWrapper.add
  websocket_impl.dart 333   _WebSocketProtocolTransformer._messageFrameEnd
  websocket_impl.dart 228   _WebSocketProtocolTransformer.add

00:00 +0 -1: Some tests failed.                                                                    

[abarth]

So, what fixed it for me was to add a dependency on flutter_test in my pubspec.yaml:

$ cat pubspec.yaml 
name: foo
description: A new flutter project.
dependencies:
  flutter:
    sdk: flutter
  flutter_test:
    sdk: flutter

Let me know if that fixes it for you. Obviously, we need to handle this failure mode more gracefully. 😸

[eseidelGoogle]

We have some known issues in our Isolate handling which affect our package:test integration. I'm hopeful those will be sussed out and fixed as part of #3443.

[Hixie]

Wow, that is a heck of an obscure way for us to fail.