AnnaBot - Annotation Assertion Based Object Testing
Status - Caveat!
I worked on this quite a few years ago, and have just started getting back into it. It’s been converted to Maven, and lightly modernized. Nothing actually works as of this commit, but it’s all moving in the right direction…
Maven + Quick Demo
For a quick demo, run:
mvn package assembly:single java -jar target/annabot-1.0.0-SNAPSHOT-jar-with-dependencies.jar abclaims target/classes
If the mvn fails to start, with a message about antlr-3.1.1, run the Unix script
scripts/install_antlr_runtime and try again.
Expected output is something like the following (filenames have changed):
Using 7 claims from directory abclaims Checking nnn targets from directory systests AnnaBot: Initialization took 0.1 seconds ... chatter about non-relevant classes ... class violations.JPAEntityNoArgConstructor: JPA Entities must have a no-arg constructor class violations.JPAEntityMethodFieldAnnotated: JPA Entity must have an ID class violations.JPAEntityMethodFieldAnnotated: JPA Entities may have field OR method annotations, not both class violations.JPAEntityHasId: JPA Entity must have an ID class violations.JPAEntitySetMethodAnnotated: JPA Entities may have field OR method annotations, not both AnnaBot: found 5 error(s) in 4 classes, time 0.3 seconds
Actual output is that it doesn’t recognize the claims files; presumably they need to be run through the compiler.
Ant + Quick Demo
For a quick demo, run the Unix script
scripts/bootstrap_ant, then do
Expected output is similar to the Maven example above.
Actual output is that Antlr fails as it is looking in the wrong directory.
Bug the Antlr step creates a bunch of temporary files in
src/main/antlr3/annabot/antlr/ that should be in a
directory, or be git-ignored.
Running on Eclipse (out of date!)
The git module "annabot" was the root of an Eclipse project. This has NOT been tested since the conversion to Maven; see http://blog.wakatta.jp/blog/2012/01/14/antlr3-maven-plugin-eclipse-setup/ for some pointers on setting up Antlr+Maven+Eclipse. The following documents the old structure, which no longer exists:
The setup is a bit unusual: there are several source folders AND several output folders:
Input Output ----- ------ src,regress build javaclaims javaclaimclasses dslclaims - systests systestclasses
To set up for developing and testing AnnaBot as an Eclipse project, assuming you checked it out under Eclipse as a project:
1) Create a new JUnit 4 Run Configuration:
Name: AnnaBot Unit Tests Class: AnnaBotTests
Run this; it should run > 50 tests and give a green bar.
2) Create a Java Application Configuration
Name: AnnaBot System Tests Class: annabot.AnnaBot Arguments -> Application Arguments: claimclasses systestclasses
Expected output: it should give output something like the Maven example above.
Actual output: fails to compile.
Now you can dive in and start working on the code!
See TODO file for what needs doing.
git update periodically. Send git pull requests, or git diff -u (or Eclipse patch) diffs.
NetBeans, IntelliJ, etc.
If you want to make it work on these, nobody will stop you. Better if you can send these IDE’s various befuddling config files, (as long as they don’t embed any full paths or otherwise break use of Maven (and Eclipse)).
Writing New Claims
This is how AnnaBot will grow to become useful, so please dive in!
Each Claim class has these required methods:
/* return the description of the problem this claim finds */ public abstract String getDescription(); /** Return a list of Operators that are the "if" part of * a claim; if any of these operators fails, the class * will be bypassed and considered successful (i.e., will not * generate any failures!) for this claim. */ public abstract Operator getClassFilter(); /** Return a list of Operators, all of which must return * true for this class to be considered successful against * this claim. * Only called if the Operators in getClassFilter all pass. */ public abstract Operator getOperators();
Examine the existing claims folder for some examples. Please contribute back to the project any claims that you develop for public APIs - thank you!
Testing the tester
A testing tool must as a matter of course be tested itself, to provide a chain of trust. The individual elements, such as the tree package’s And, AtMostOne, and friends, are all tested routinely using Junit. These will be verified by "mvn package" automatically invoking the standard JUnit4 test runner on the class tests found in the src/test folder. This also provides some tests for the basic functionality of the AnnaBot control program.
In addition, there are integration tests showing that AnnaBot does detect the failure modes described by the Claim files for JPA; these are available in the systests folder of the project.
Given these tests, I did not immediately see a need to do permutation testing on a body of existing code, since the Integration Tests provided with the package will provide an assurance that the package will find erroneous annotations it they exist. An anonymous reviewer convinced me that mutation testing is regarded as a standard approach. Manually introducing several errors into one of the sample codebases (the Toronto Centre for Phenogenomics, see Section 4) and re-running the program ensured that the errors were correctly detected.
Tests of the ClassSource files (used to discover classes in Jar files and in directory hierarchies) are in the author’s darwinsys-api project, available in a jar file from Maven Central and in source code from the same github account as AnnaBot itself.