AnnaBot - Annotation Assertion Based Object Testing
Ian Darwin, http://www.darwinsys.com/
For a quick demo, run
java -classpath build:lib/javaee.jar:lib/darwinsys.jar annabot.AnnaBot claimclasses systestclasses
RUNNING ON ECLIPSE
The git module "annabot" is the root of an Eclipse project. The setup is a bit unusual: there are several source folders AND several output folders:
Input Output ----- ------ src,regress build javaclaims javaclaimclasses dslclaims - systests systestclasses
To set up for developing and testing AnnaBot as an Eclipse project, assuming you checked it out under Eclipse as a project:
1) Create a new JUnit 4 Run Configuration:
Name: AnnaBot Unit Tests Class: AnnaBotTests
Run this; it should run > 50 tests and give a green bar.
2) Create a Java Application Configuration
Name: AnnaBot System Tests Class: annabot.AnnaBot Arguments -> Application Arguments: claimclasses systestclasses
Run this; it should give output something like the following:
Using 7 claims from directory claimclasses Checking 4 targets from directory systestclasses AnnaBot: Initialization took 0.1 seconds class violations.JPAEntityNoArgConstructor: JPA Entities must have a no-arg constructor class violations.JPAEntityMethodFieldAnnotated: JPA Entity must have an ID class violations.JPAEntityMethodFieldAnnotated: JPA Entities may have field OR method annotations, not both class violations.JPAEntityHasId: JPA Entity must have an ID class violations.JPAEntitySetMethodAnnotated: JPA Entities may have field OR method annotations, not both AnnaBot: found 5 error(s) in 4 classes, time 0.3 seconds
Now you can dive in and start working on the code!
Remember: git update periodically. Send git diff -u (or Eclipse patch) diffs.
If you want it, nobody will stop you. Better if you can send a "nbproject" folder (as long as it doesn’t embed any full paths!).
There is the outline of an Ant task but it does not work as of this writing. Anyway, there needs to be two tasks, AnnaBotC and AnnaBot.
You should be able to:
jar up the "build" folder, add that jar to another project's classpath; jar up the "claimclasses" but don't add that jar to classpath run AnnaBot with the claimclasses.jar and the target classes.
WRITING NEW CLAIMS
Each Claim class has three required methods:
/* return the description of the problem this claim finds */ public abstract String getDescription();
/** Return a list of Operators that are the "if" part of * a claim; if any of these operators fails, the class * will be bypassed and considered successful for this claim. */ public abstract Operator getClassFilter();
/** Return a list of Operators, all of which must return * true for this class to be considered successful against * this claim. * Only called if the Operators in getClassFilter all pass. */ public abstract Operator getOperators();
Examine the existing claims folder for some examples. Please contribute back to the project any claims that you develop for public APIs - thank you!
Testing the tester
A testing tool must itself be tested, to provide a chain of trust. The individual elements, such as the tree package’s And, AtMostOne, and friends, are all tested routinely using Junit [TestInfected]. These can be independently verified by invoking any standard JUnit4 test runner on the class tree.AllTests found in the regress folder. This is invoked automatically from the AnnaBotTests class found in the root of the regress folder; this also provides some tests for the basic functionality of the AnnaBot control program.
In addition, there are integration tests showing that AnnaBot does detect the failure modes described by the Claim files for JPA; these are available in the systests folder of the project.
Given these tests, I did not immediately see a need to do permutation testing on a body of existing code, since the Integration Tests provided with the package will provide an assurance that the package will find erroneous annotations it they exist. An anonymous reviewer convinced me that mutation testing is regarded as a standard approach. Manually introducing several errors into one of the sample codebases (the Toronto Centre for Phenogenomics, see Section 4) and re-running the program ensured that the errors were correctly detected.
Tests of the ClassSource files (used to discover classes in Jar files and in directory hierarchies) are in the author’s darwinsys-api project, available from the same git repository as AnnaBot itself.
There is an experimental Antlr grammar, but it has not been worked on in several years and, truth to tell, I have no idea what state it’s in. Just committed here in case somebody wants to run with it.