Allow skipping on a test-by-test basis

[matthew-carroll]

To deal with developers on different Mac OSes, I'm writing goldens to different directories based on the OS version:

goldens_macos_10.15/
goldens_macos_11/

With different developers generating these goldens at different times, we'd like to skip golden tests when the golden file doesn't exist yet. We don't want that triggering a failure.

We'd like an opportunity to control that skipping behavior. This could either come as a configuration to skip tests that don't have a file, or it might make more sense to provide a per-test callback where the callback is given the golden file name and then decides whether or not to run the goldens comparison.

[coreysprague]

Trying to get clarity on your scenario...

Is it something like this?

• developer A adds a new testGoldens() test in file_a.dart, generates goldens on macOS_11
• developer B adds a new testGoldens() test in file_b.dart, generates goldens on macOS_10.15

the developers merge in their changes, and you want developer A's flutter test to pass despite missing a golden file for file_b.dart, and developer B's flutter test to pass despite missing a golden file for file_a.dart?

[matthew-carroll]

@coreysprague that's correct. And the same is true for CI. Failures in CI is the primary outstanding problem, but that problem is the same as what you've described.

[matthew-carroll]

Also, another feature that might make sense with this change would be to only update goldens that don't exist yet. That way, I can checkout a repo and immediately generate any goldens that don't exist for my platform yet, without worrying about overwriting the existing goldens for my platform.