Support using lightweight Flutter Engines to run tests

[derekxu16]

This PR implements the functionality described above and hides it behind the --experimental-faster-testing flag of flutter test.

The following are some performance measurements from test runs conducted on GitHub Actions

run 1 logs: https://github.com/derekxu16/flutter_test_ci/actions/runs/8008029772/attempts/1
run 2 logs: https://github.com/derekxu16/flutter_test_ci/actions/runs/8008029772/attempts/2
run 3 logs: https://github.com/derekxu16/flutter_test_ci/actions/runs/8008029772/attempts/3

length of flutter test --reporter=expanded test/animation test/foundation step

run 1: 54s
run 2: 52s
run 3: 56s

average: 54s

length of flutter test --experimental-faster-testing --reporter=expanded test/animation test/foundation step

run 1: 27s
run 2: 27s
run 3: 29s

average: 27.67s (~48.77% shorter than 54s)

length of flutter test --reporter=expanded test/animation test/foundation test/gestures test/painting test/physics test/rendering test/scheduler test/semantics test/services step

run 1: 260s
run 2: 270s
run 3: 305s

average: 278.33s

length of flutter test --experimental-faster-testing --reporter=expanded test/animation test/foundation test/gestures test/painting test/physics test/rendering test/scheduler test/semantics test/services step

from a clean build (right after deleting the build folder):

run 1: 215s
run 2: 227s
run 3: 245s

average: 229s (~17.72% shorter than 278.33s)

Note that in reality, test/material was not passed to flutter test in the trials below. All of the test files under test/material except for test/material/icons_test.dart were listed out individually

length of flutter test --reporter=expanded test/material step

run 1: 408s
run 2: 421s
run 3: 451s

average: 426.67s

length of flutter test --experimental-faster-testing --reporter=expanded test/material step

run 1: 382s
run 2: 373s
run 3: 400s

average: 385s (~9.77% shorter than 426.67s)

[derekxu16]

cc @a-siva

[derekxu16]

I just tried running flutter test --experimental-faster-testing on Windows to investigate the Windows failures. The problem is that when running root_test_isolate_spawner.dart, the _loadLibraryFromKernel call here

https://github.com/flutter/flutter/blob/3380abbb68a1b84d0c17823602fc5f632ce60925/packages/flutter_tools/lib/src/test/runner.dart#L519-L521

is returning null, giving the error:

[ERROR:flutter/shell/testing/tester_main.cc(666)] Unhandled exception
Exception: type 'Null' is not a subtype of type '(SendPort) => void' in type cast
Stack trace: #0      main (file:///C:/Users/Derek/flutter/packages/flutter/build/isolate_spawning_tester/root_test_isolate_spawner.dart:70:32)

I tried changing '${childTestIsolateSpawnerDillFile.absolute.path}' to 'file://${childTestIsolateSpawnerDillFile.absolute.path}', but that had no effect. Any ideas @dnfield?

[dnfield]

The windows version of this needs a windows-style path. File URIs won't work.

[derekxu16]

The checks are now all passing.

[dnfield]

What's the result of using this flag locally to run tests compared to the "default" behavior?

What's the plan for implementing this in Flutter CI?

[dnfield]

This LGTM to get things rolling. I'd appreciate it if you remove the comment I tagged in the pubspec. It also needs a merge conflict fix.

It would be very helpful if you add some numbers even if just locally to this PR to compare how it's running on e.g. the framework tests with and without the flag.

[derekxu16]

Running the material tests with --experimental-faster-testing produces some Golden pixel test failures that don't occur when running without the flag. I'm trying to fix these now.