The tool integration tests are leaky and need significant work to fix

[dnfield]

This may need to be split into separate issues, but while I still have the context in my head I want to write this out.

See also #51382

Tests in the ingtegration.shard folder tend to have the following structure:

• Launch Flutter via the shell script entrypoint (flutter.bat or flutter depending on the OS).
• This might be done multiple times, e.g. to create a project in a temp dir, and then to launch flutter test --start-paused.
• This causes Flutter to spawn a Dart VM process that is told to never terminate.
• The test tries to clean things up by connecting to the VM, getting its PID, and eventually killing the process
• killPid in Dart does not guarantee process death, and does not wait for the process to terminate
• Killing the shell script invocation does not necessarily kill the launched flutter_tester process, or, if it does, it takes longer than the termination of the shell script itself.
• The test attempts to delete temporary files, and swallows exceptions. A comment in the source indicates we believe this to be a permissions issue, where in reality it's that we spawned a process that hasn't terminated that is holding an exclusive lock on the files (on Windows).

I attempted to deflake some of this in the linked PR, but my attempts are still flakey because we don't have a handle on the Process to be able to actually wait for it to exit.

We could potentially refactor these tests to not use the flutter/flutter.bat entrypoints so that we can properly control process death. We should also investigate whether flutter test --start-paused leaks processes on Windows.

/cc @zanderso @Hixie @jonahwilliams

[dnfield]

Here's a simple way to exercise what I think is a core bug here:

on Windows, run the following:

flutter create testapp
cd testapp
flutter pub get
flutter test --start-paused test\widget_test.dart

Wait for the tester to get up and running, press ^C, and agree to terminate the batch script.

flutter_tester.exe will die, but if you look in %TEMP%, you're left with the folders that the test harness created - presumably because it failed to actually wait for the process to die.

[dnfield]

I've tried adding a signal handler to flutter_platform.dart to ensure the finalizers collection runs on Ctrl+C if possible. It hangs at trying to kill the flutter_tester process.

[jonahwilliams]

The tool integration tests aren't flaky though, they're just leaky.

[dnfield]

s/flaky/leaky

[dnfield]

And really, the only reason they're not flaky is because we swallow exceptions related to the leaks, and so if the leak is bad enough we still pass even though we should really fail.