Restructure analyzer test cases to be null safe by default

[srawlins]

I don't know if anyone has vigorously updated our test files to include both a pre-null-safety and a post-null-safety class, even if there are no new or deprecated test cases for the new class. For example, non_constant_case_expression_test.dart contains:

main() {
  defineReflectiveSuite(() {
    defineReflectiveTests(NonConstantCaseExpressionTest);
    defineReflectiveTests(NonConstantCaseExpressionWithNullSafetyTest);
  });
}

@reflectiveTest
class NonConstantCaseExpressionTest extends PubPackageResolutionTest {
  // Test cases
}

@reflectiveTest
class NonConstantCaseExpressionWithNullSafetyTest
    extends NonConstantCaseExpressionTest with WithNullSafetyMixin {}

I'm happy to grind through "all" of our test files (maybe all diagnostics/ tests and a few other directories?) and add null safety sibling test classes.

In addition, maybe I'll take the step of defaulting to null safety (since it is the default language version now), and write classes like FooTest and FooWithoutNullSafetyTest.

[srawlins]

CC @bwilkerson @scheglov

[bwilkerson]

I'll defer to Konstantin on this one, but we discussed doing just this when we started implementing null-safety and the consensus at that point is that we didn't need to duplicate the number of tests when they generally didn't depend on null-safety semantics. I think that was and still is the right decision.

Whether we want to reverse the names (and order in the hierarchy) of the test classes that have a WithNullSafety companion class is a whole other question. While it would keep the naming consistent moving into the future, it sounds like a lot of work with a questionable amount of benefit. I'd be inclined to leave it as is until the naming starts to become a burden.

[srawlins]

... we didn't need to duplicate the number of tests when they generally didn't depend on null-safety semantics. I think that was and still is the right decision.

That makes sense to me. What about just default turning on null safety in all test classes that don't have a sibling WithNullSafety class?

[scheglov]

We were adding WithNullSafety tests when there is a difference in resolving legacy and null safe code.
When there is no difference, it would be fine to run with null safety.
I have a migration CL for the analyzer, and it would be better if we avoided doing large scale changes before landing this CL to reduce amount of rebase.

[srawlins]

I have a migration CL for the analyzer, and it would be better if we avoided doing large scale changes before landing this CL to reduce amount of rebase.

Haha don't worry, I'm keeping any lower-priority work in a holding pattern until that CL lands. Easier for me to rebase a small CL on a giant change than vice versa.

[scheglov]

One approach to reduce the number of combinations could be that the base of all tests (usually PubPackageResolutionTest) always uses the latest language version, and also all experimental features enabled. And then when we don't want some experiments - we write @dart = x.y in files of this test; or do writeTestPackageConfig.

Some tests we might want to run in both before-null-safety and such latest language with experiments modes. In this case we can continue to have two classes and XyzTestCases mixins.

[srawlins]

I think this sounds good.

I doubt we will re-write the type system too many more times, and for smaller language features like non-function typedefs, triple-shift, annotations-with-type-args, etc, the vast majority of tests should work with and without the language feature. For the tests which are affected, individual care should be taken, such as multiple test classes.