Support for static diagnostic tests for macros

[jakemac53]

Macros can emit diagnostics, of varying severity. In the SDK repo, we have language tests for validating diagnostics that we expect compilers to report, and I think they generally work fairly well.

It would be interesting to explore a similar functionality, but for macro authors, and potentially analyzer plugin authors as well.

I haven't put much thought into how they would work exactly, but possibly some metadata could be used to annotate a test as not being executable, and instead being a "static diagnostic test". And we would either re-use the existing syntax for the SDK tests or invent something with similar functionality.

WDYT @natebosch ?

[natebosch]

We might want to build this as a separate package. We could start with a manual integration with the test runner.

// test/diagnostics_test.dart
import 'package:diagnostics_tests/diagnostic_tests.dart';

void main() {
  runDiagnosticsTests('path/to/directory');
}

runDiagnosticsTests would call group and test to define appropriate test cases based on the files it reads.

After we are confident the API is stable we could consider integrating it into package:test more directly.

[jakemac53]

A different package, at least to start, I think makes sense.

Another thing to consider here is that diagnostics can be emitted inside of macro generated code too, so we would have to figure out how that should be expressed.

[jakemac53]

Another note here is that the way existing language tests work can be frustrating because your IDE will also report all the errors, for all the tests, in the problems view.

It is useful when looking at the actual error test to see the actual diagnostics in line, but it can also clutter the problems view a lot. It would be nice to somehow get errors inline, but not in the problems view, and also when running dart analyze . on the project not have the errors reported.

[natebosch]

It would be nice to somehow get errors inline, but not in the problems view

I don't think this is feasible. IIRC the server gives a list of diagnostics, but doesn't control how the IDE displays them in the source or other views. I don't think we can mark a diagnostic to excluded it from the problems view.

[bwilkerson]

... and potentially analyzer plugin authors as well.

I'm expecting that we'll have a test framework for plugin authors that looks more like the style of tests that we use in the analysis server. While something like this might be sufficient for diagnostics, I don't think it would be a great fit for testing other IDE features, such as quick fixes.