Custom JUnit TestRunner

[JeffFaer]

The current pattern for tests leaves something to be desired:

• Each *(Positive|Negative)Cases.java file generally contains several checks. It would be nice if each check was an individual test case (@Test) so that it is easier to see exactly what is failing.
• Many *Test.java classes contain Stringly typed source code. It's a pain in the neck to format it and add imports.
• This isn't Java 4 anymore, we have annotations! We could use an annotation instead of a comment to declare where we think errors are.
  • Advantages:
    • Easier to see what properties we can assert.
    • Facilitates better documentation
  • Disadvantages:
    • We have to create a new class for the annotation
    • It is harder to match an annotation type if you're only given a String. But we do have access to the AST...

My proposal:

• Create a TestRunner which will report each expected diagnostic as a separate test case with a specific name. Let's call it ErrorProneRunner.
• Annotate each *(Positive|Negative)Cases.java file with @RunWith(ErrorProneRunner.class)
• Create dummy packages in testdata for Stringly typed source code and tell the ErrorProneRunner to do its thing (possibly with an annotation on *PositiveCases.java?)
• Replace // BUG: Diagnostic contains: XXX with @ExpectDiagnostic("specific expectation message"):

@interface ExpectDiagnostic {
    String value();
    String[] messageContains() default {};
    String[] fixesContain() default {};
    .
    .
}

• We can also add the counterpoint @DoNotExpectDiagnostic

Thoughts?

[alexeagle]

Sorry for the long delay. That seems pretty reasonable to me, except for a concern about placing an annotation in every place we currently have the //BUG comment - are all of those places legal to place an annotation?
Are you interested in contributing this still?

[tjog]

Hi, we (a group part of a software engineering course at KTH) are going to try and tackle some of this issue. The sheer amount of files this affects makes it difficult to estimate how much we will accomplish, in terms of reducing Stringly typed source code test cases and annotating with @Test in Cases.java files, but the TestRunner and ExpectDiagnostic components will hopefully be implemented and then start the conversion.

[tjog]

So after working on this our group concludes, like the above comment suggests, annotations are not general enough to target all locations currently targeted by the // BUG comments. Further, we find the solution of annotating **Cases.java files with @RunWith(...) as contradicting the purpose of those files, as they are really only resources. The intermediate test file that "includes" the Cases files is the one which should have the custom runner annotation.

We are out of time working on this issue, but if someone were to develop this further we would revise the plan to:

• Create a custom runner that extends a default runner to allow the ordinary jUnit @Test to function like normal, but also operate on a new test annotation like @MultiTest that incorporates some of the test diagnostics and compilation helpers code to add // BUG occurences dynamically to the number of tests run for the overall test class.
• The @MultiTest annotation could set some mode inside the helpers that do this, as having an argument @MultiTest("FileWithCases.java") may not be flexible enough.
• Start switching over the test files which use cases file to the @MultiTest annotation over the standard @Test.