☂️ Improved Test Ergonomics

[chgibb]

Hydro-SDK's current testing workflow forces running widget tests for entire Hydro-SDK components at a time. There is no built in method to allow unit tests. There is no built in method to allow widget tests that are not conducted on an entire Hydro-SDK component.

Hydro-SDK needs a built in testing workflow that is analogous to the workflow enjoyed by users of flutter test / jest.
In general, for every file matching a given pattern;

• Build it
• Execute it in an environment providing a headless implementation of dart:ui (like flutter_tester)

Code in Hydro-SDK is organized around projects and components. Each test file should be built as a unique synthetic component of the current project. This will allow existing build tooling to continue to safely assume that every component build is fully independent.

In Flutter, the flutter_tester executable is invoked with the path to a kernel file compiled from the file to be tested. For Hydro-SDK, flutter_tester should be able to be copied and distributed (as opposed to building it from source). Hydro-SDK can distribute a test harness kernel file that will load and run a specified .ota package from the filesystem. The test harness should provide a CFR with dart:*, package:flutter, package:flutter_test and package:spec APIs available.

In general, there are two unique scenarios where a new test workflow will be consumed;

1. Downstream use of Hydro-SDK
2. Upstream hacking on Hydro-SDK directly

For downstream use, requiring the user to have a Dart and Flutter installation to use Hydro-SDK should be eliminated as a requirement. A new test workflow should come out of the box with everything needed to run unit, widget, and golden tests.

For hacking on Hydro-SDK, a creaky system of compiling and directly running bytecode chunks with a series of shell scripts and flutter test invocations has been employed until now. The same Dart code distributed as pre-compiled kernel in the downstream use case should be able to be used upstream to test Hydro-SDK directly. This should allow for better debuggability of upstream tests. This same Dart code should also be made available as a library to downstream users who may wish to customize their test experience (like with custom namespaces or other configuration).

Out of scope

• Connecting a debugger to a Hydro-SDK test running in the downstream usecase
• Inconsistencies running in Dart debug vs profile vs release mode. If inconsistencies are encountered (especially around the use of kDebugMode, kReleaseMode and Dart's built in assert, some of the work under "Recovering framework asserts" from ☂️ Component Preview #763 might have to be considered

Open questions

• Asset directory for flutter_tester
• ICU data for flutter_tester
• Using an AOT compiled test harness instead of kernel
• Changes to hydroc to support distributing platform kernel files in addition to platform executables