[material_ui, cupertino_ui, go_router] Command to test dart fixes by justinmc · Pull Request #12390 · flutter/packages

justinmc · 2026-08-06T23:37:18Z

This PR creates a new repo-level CI command to run the dart fix tests in relevant packages. Before this PR, go_router had its own script to run its dart fix tests, and material_ui and cupertino_ui had dart fix tests that were not being run. After this PR, all of the above are run via the new command.

Part of flutter/flutter#182568

Adapted from go_router's run_tests script.

gemini-code-assist

Code Review

This pull request replaces the standalone tool/run_tests.dart script with a new test-dart-fixes command in the repository tooling, implemented in TestDartFixes. This command automates running Dart fix tests for packages containing a test_fixes directory. Feedback on the implementation highlights opportunities to improve testability by using the package's file system and the configured processRunner instead of hardcoded LocalFileSystem and Process.start calls. Additionally, minor typos in string interpolation within error messages were identified.

gemini-code-assist · 2026-08-07T19:53:42Z

+  static Future<int> _runProcess(
+    String command,
+    List<String> arguments, {
+    String? workingDirectory,
+  }) async {
+    final Process process = await _streamOutput(
+      Process.start(command, arguments, workingDirectory: workingDirectory),
+    );
+    return process.exitCode;
+  }


Calling Process.start directly from dart:io bypasses the processRunner passed to the constructor, which prevents mocking process execution in unit tests. Consider making these helper methods non-static and using processRunner to run the processes.

gemini-code-assist · 2026-08-07T19:53:42Z

+    try {
+      testDirectory = await _createTestDirectory(package);
+    } catch (error) {
+      return PackageResult.fail(['Failed to create temporary test directory: $error}']);


There is a typo in the string interpolation: $error} has an extra closing brace. It should be $error.

Suggested change

return PackageResult.fail(['Failed to create temporary test directory: $error}']);

return PackageResult.fail(['Failed to create temporary test directory: $error']);

gemini-code-assist · 2026-08-07T19:53:42Z

+      }
+      result = PackageResult.success();
+    } catch (error) {
+      result = PackageResult.fail(['Dart fix tests failed: $error}']);


There is a typo in the string interpolation: $error} has an extra closing brace. It should be $error.

Suggested change

result = PackageResult.fail(['Dart fix tests failed: $error}']);

result = PackageResult.fail(['Dart fix tests failed: $error']);

justinmc · 2026-08-07T19:54:32Z

Questions for @stuartmorgan-g in code review:

There are only 3 packages that support this, so it leaves a bunch of spam in the terminal saying "skipped x package". Is that ok or is there a way I should be avoiding that?
I deleted go_router's script that did this previously, but is there anything else that I should clean up there?

stuartmorgan-g

Two main high-level things

This command needs to be added to the CI so it's actually running. How long does it take to run? If it's fast we could just add it to https://github.com/flutter/packages/blob/main/.ci/targets/dart_unit_tests.yaml. If it's not, we'll need to think about where we can add it without causing issues.
We unit test all of the repo tooling, so this needs corresponding tests in script/tool's test directory. See other commands for the common patterns of DI, faking process output/exit codes, etc., and let me know if you have any questions.

There are only 3 packages that support this, so it leaves a bunch of spam in the terminal saying "skipped x package". Is that ok or is there a way I should be avoiding that?

That's a feature :)

(For context, when I started working on this repo, I discovered that we had a significant number of tests that people thought we were running but weren't because of various mistakes that were silently ignored, and it was extremely hard to figure that out because the CI output was just a sea of results for the things that did run. As a result, the tool is now designed for auditability, so that it's very easy to look at a run log and see exactly what did and did not run, to make sure that matches expectations.)

I deleted go_router's script that did this previously, but is there anything else that I should clean up there?

I don't think so; removing run_tests.dart will cause the package to be skipped by the custom-tests command (in the same way that removing test_fixes would cause a package to be skipped by this new command).

stuartmorgan-g · 2026-08-12T15:31:21Z

+      'This command requires "flutter" to be in your path.';
+
+  @override
+  PackageLoopingType get packageLoopingType => PackageLoopingType.includeAllSubpackages;


Could sub-packages ever have their own dart fixes? If not we only need to look at top-level packages.

It's possible but doesn't happen in any of our packages. I'll set it to .topLevelOnly.

stuartmorgan-g · 2026-08-12T15:32:33Z

+      return PackageResult.fail(['Failed to create temporary test directory: $error']);
+    }
+
+    late final PackageResult result;


Does this actually need the late keyword? I would expect it to work without it (with better safety).

stuartmorgan-g · 2026-08-12T15:36:15Z

+  /// Run the dart fix tests for the package in the given temporary directory.
+  ///
+  /// Resolves with the status code of the command.
+  Future<int> _runDartFixTests(RepositoryPackage package, Directory testDirectory) async {


package is never used.

stuartmorgan-g · 2026-08-12T15:37:19Z

+    final int pubGetStatusCode = await _runProcess('dart', <String>[
+      'pub',
+      'get',
+    ], workingDirectory: testDirectory);


We have runPubGet to abstract this, in pub_utils.dart. You just need to construct a RepositoryPackage from the test directory.

stuartmorgan-g · 2026-08-12T15:39:22Z

+    ], workingDirectory: testDirectory);
+  }
+
+  Future<int> _runProcess(


These methods should be removed in favor of the command taking a ProcessRunner, and using processRunner.runAndStream. See dart_test_command.dart for an example (or almost any other command).

… is not too long

justinmc · 2026-08-12T23:03:17Z

How long does it take to run?

About 40 seconds. Most of the time is from running flutter pub get three times in series. I've added it to dart_unit_tests.yaml preemptively, but let me know if that's too long and we should put it somewhere else.

justinmc · 2026-08-12T23:59:46Z

CI is failing and some of the failures have to do with my new tests, but the error messages are cryptic. I'm going to push a merge commit and come back to this tomorrow.

stuartmorgan-g · 2026-08-26T23:59:28Z

+  final String name = 'test-dart-fixes';
+
+  @override
+  List<String> get aliases => <String>[];


This is the default value, so no override is needed.

stuartmorgan-g · 2026-08-26T23:59:33Z

+      'This command requires "flutter" to be in your path.';
+
+  @override
+  PackageLoopingType get packageLoopingType => PackageLoopingType.topLevelOnly;


stuartmorgan-g · 2026-08-27T00:08:47Z

+    try {
+      final bool success = await _runDartFixTests(testDirectory);
+      if (!success) {
+        throw Exception('Failed to run dart fix tests.');


Rather than turning a bool into an exception, why not just:

result = success ? PackageResult.success() : PackageResult.fail(['Tests failed']);

It seems easier to follow, and also avoids the common failure case having the string "Dart fix tests failed: Failed to run dart fix tests." (which seems confusing).

Agreed, good catch.

stuartmorgan-g · 2026-08-27T00:11:07Z

+    final bool success = await runPubGet(RepositoryPackage(testDirectory), processRunner, platform);
+
+    if (!success) {
+      return success;


Actually, if you make this method return a PackageResult rather than a bool, you could make the error summary distinguish between pub get failing and the tests themselves failing.

Thanks. I ended up making it throw an exception, but same idea.

stuartmorgan-g · 2026-08-27T00:12:54Z

+          contains('Running for package1'),
+          contains('Running for package2'),
+          contains('Running for package3'),
+          contains("SKIPPING: No MemoryDirectory: '/packages/package3/test_fixes' directory."),


This weird error message is because you are logging the file system object rather than its path; we should just have the path (preferably the package-relative path rather than the absolute path).

stuartmorgan-g · 2026-08-27T00:16:20Z

+  });
+}
+
+/// Fails when fix.dart does not equal fix.dart.expect.


What's the purpose of simulating the internal behavior of the fix tests? The command itself doesn't appear to know anything about the way the test works internally, so it seems like you'd get just as much test value out of just doing our standard exit value mocking, without the complexity of this class and of creating fake test-only expectation files just for this test-only code to read them to decide to fail.

You're right, simplified in 190f200.

Remove unnecessary overrides, and refactor _runDartFixTests.

chunhtai

LGTM

stuartmorgan-g

LGTM

…r#192119) flutter/packages@d642322...7a7912f 2026-09-01 21270878+elliette@users.noreply.github.com [material_ui] Migrate AppBar M3 template to use new gen_defaults script (flutter/packages#12707) 2026-09-01 32538273+ValentinVignal@users.noreply.github.com [vector_graphics] Fix deprecation lints (flutter/packages#12702) 2026-08-31 21270878+elliette@users.noreply.github.com [material_ui] Standardize M3 and M3E token template base class names (flutter/packages#12675) 2026-08-31 32538273+ValentinVignal@users.noreply.github.com [vector_graphics_compiler] Fix deprecation issues in tests (flutter/packages#12700) 2026-08-31 stuartmorgan@google.com [google_maps_flutter] Convert marker controllers to Swift (flutter/packages#12662) 2026-08-31 jmccandless@google.com [material_ui, cupertino_ui, go_router] Command to test dart fixes (flutter/packages#12390) 2026-08-31 engine-flutter-autoroll@skia.org Roll Flutter from c978386 to c8c5e3b (4 revisions) (flutter/packages#12706) 2026-08-31 269567208+reidbaker-agent@users.noreply.github.com [tool] Exempt AGENTS.md from version and changelog checks (flutter/packages#12681) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Barebones command copied from the test command.

de72c52

justinmc self-assigned this Aug 6, 2026

justinmc added 2 commits August 7, 2026 12:34

Works for running dart fix tests on go_router, cupertino, and material.

dfa7553

Adapted from go_router's run_tests script.

This is covered by the global script now.

e280734

github-actions Bot added p: go_router triage-framework Should be looked at in framework triage labels Aug 7, 2026

Code cleanup

20b7842

justinmc added the CICD Run CI/CD label Aug 7, 2026

justinmc requested a review from stuartmorgan-g August 7, 2026 19:52

justinmc marked this pull request as ready for review August 7, 2026 19:52

gemini-code-assist Bot reviewed Aug 7, 2026

View reviewed changes

Gemini code review

52517d3

justinmc mentioned this pull request Aug 7, 2026

Publish (as publicly listed) the Cupertino and Material pub packages flutter/flutter#182568

Closed

3 tasks

stuartmorgan-g requested changes Aug 12, 2026

View reviewed changes

justinmc added 7 commits August 12, 2026 13:59

flutter pub get, not dart

9167024

Add to CI unit tests, under the assumption that the ~40s running time…

f7232b7

… is not too long

Command in name

de6d383

Copy in memory temp directory properly.

08a076f

Variable name consistency

9173808

Working tests

1e58804

Fixes from stuart's code review

e553f85

justinmc added 2 commits August 12, 2026 16:06

Unused import

a9cbde2

Merge branch 'main' into test-dart-fixes

9e6810a

justinmc added 3 commits August 13, 2026 09:27

Fix CI execution of dart fix tests

94e7a11

Fix for Windows by using FileSystem not Path

aa09ca3

Formatting

143f2b9

justinmc requested a review from stuartmorgan-g August 13, 2026 17:54

stuartmorgan-g reviewed Aug 27, 2026

View reviewed changes

justinmc added 4 commits August 27, 2026 15:22

Code clean up from stuart's review.

3488d8a

Remove unnecessary overrides, and refactor _runDartFixTests.

Better skip error message.

030f0f3

Simplify process test.

190f200

Local path.

ab71175

github-actions Bot added the p: go_router_builder label Aug 28, 2026

Merge branch 'main' into test-dart-fixes

e6f10c3

justinmc force-pushed the test-dart-fixes branch from db400f5 to e6f10c3 Compare August 28, 2026 18:07

github-actions Bot removed the p: go_router_builder label Aug 28, 2026

Test matches new local directory output.

a3f29be

justinmc requested review from chunhtai and stuartmorgan-g August 28, 2026 22:33

chunhtai approved these changes Aug 28, 2026

View reviewed changes

stuartmorgan-g approved these changes Aug 31, 2026

View reviewed changes

justinmc added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 31, 2026

auto-submit Bot merged commit c8faec1 into flutter:main Aug 31, 2026
13 checks passed

justinmc deleted the test-dart-fixes branch August 31, 2026 17:09

This was referenced Sep 1, 2026

Roll Packages from d642322f51b5 to 2e33644f5943 (7 revisions) flutter/flutter#192113

Closed

Roll Packages from d642322f51b5 to 7a7912faa4dd (8 revisions) flutter/flutter#192119

Merged

	return PackageResult.fail(['Failed to create temporary test directory: $error}']);
	return PackageResult.fail(['Failed to create temporary test directory: $error']);

	result = PackageResult.fail(['Dart fix tests failed: $error}']);
	result = PackageResult.fail(['Dart fix tests failed: $error']);

Uh oh!

Conversation

justinmc commented Aug 6, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

justinmc commented Aug 7, 2026

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinmc commented Aug 12, 2026

Uh oh!

justinmc commented Aug 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chunhtai left a comment

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants