-
Notifications
You must be signed in to change notification settings - Fork 1
Atomicity detection and regression test generation through Feedback driven random test generation
Writing concurrency tests is enormously difficult: aside from the lack of expressiveness around parallelism and concurrency in popular testing frameworks, finding and laying-out the setup required to push a program into a state where it may violate atomicity rules is difficult. This makes a completely automated detection and generation-of-tests scheme for such a problem very attractive.
[problems with writing concurrent code. Need for automated test generation]
[comment on discussion section
[randoops goal; reasonably wide adoption]
[intruder -- mention that we didn't run down the quality of the atomicity violations themselves. Ultimately we took their number of 'violations' as final even if they didn't result in a bug.] One of the key insites of the intruder project is that given a set of tests that directly and minimally reproduce the access patterns for a particular object under test, many of the complexities associated with detecting atomicity problems from scratch disappear.
Our initial strategy was to simply execute randoop on the libraries from [Intruder table 5], and pipe the results from randoop into intruder
for testFramework in Colt-DynamicBin1D OpenJdk-StringBuffer #...
randoopTests = randoop --input $testFramework
compiledRandoopTests = javac $randoopTests
intruderReport = intruder --input $compiledRandoopTests
echo $intruderReport >> summary.txt[things that needed to be done to make that happen: an adapter between randoop and intruder, dependency management, build-system wet-work. ]
The single biggest technical obstacle to driving intruder with randoop generated tests was that Randoop expects the results to be driven by JUnit and Intruder expects to have a single static main entry point. Unfortunately because of the analsysis strategy of intruder we were afraid that we might see artifacts if we simply wrote a main method wrapping the JUnit API to drive the tests created by randoop. To overcome this we wrote a small java utility porter that generates a main method wrapping target junit test methods created by randoop.
Given such a utility, it was relatively easy to script the execution of randoop, the execution of porter on the results from randoop, and the execution of intruder on the results from porter.
[import problems, library problems, build problems. Build system wet-work. Ultimately correcting broken import headers was beyond the scope of this project]
[manually running tests] Table 1:
| Test | Original | Randoop |
|---|---|---|
| intruder-funcitonal-test-1 | 0 | 0 |
| intruder-funcitonal-test-2 | 1949, 1, 1 | ERROR while running intruder |
| intruder-funcitonal-test-3 | 0, | 0 |
| intruder-funcitonal-test-4 | 1838, 1, 3 | 2321, 2, 5 |
| intruder-funcitonal-test-5 | 0 | 0 |
| intruder-funcitonal-test-6 | 2216, 1, 1 | 2791, 1, 1 |
| intruder-funcitonal-test-7 | 2504, 1, 1 | Randoop Unable to Generate Test |
| intruder-funcitonal-test-8 | 2633, 1, 1 | 3242, 1, 0 |
| intruder-funcitonal-test-9 | 0 | 0 |
| intruder-funcitonal-test-10 | 0 | 0 |
| intruder-funcitonal-test-11 | 0 | 0 |
| intruder-funcitonal-test-12 | 0 | 0 |
| intruder-funcitonal-test-13 | 0 | 0 |
Colt, DynamicBin1D
|
159744, 9, 27 | 97559, 8, 19 |
Batik, CompositeGraphicsNode
|
35959, 7, 47 | 21217, 2, 2 |
Batik, CompositeGraphicsNode
|
35959, 7, 47 | 21441, 10, 7 |
[code coverage figures?]
[some general guidelines i've learned to avoid atomicity violations. Namely, ref-transparent programming, private locks.] All of the bugs we identified intruder as detecting related to failure to properly lock all of the involved functions. In the Motivation section for indruder, Figure 1 [intruder] requires the use of the mutable collection variables that are not sufficiently synchronized. This can be solved with a more elaborate use of locks or use of immutable data structures. [more op-ed?]
[randoop's coverage strategy doesn't necessarily map to atomicity discovery. Inherently it finds some, possibly the most common, violations but it does not find all of them.]
[proposal: https://l.facebook.com/l.php?u=https%3A%2F%2Fcdn.fbsbx.com%2Fhphotos-xfa1%2Fv%2Ft59.2708-21%2F12726113_10153429612116046_1992673929_n.docx%2FProposal.docx%3Foh%3D2d09b7192c51c06c4a49e0c281492cf5%26oe%3D57168EBE%26dl%3D1&h=WAQH8u-0Q] [intruder: http://drona.csa.iisc.ernet.in/~muralikrishna/publications/fse15.pdf] [randoop: http://homes.cs.washington.edu/~mernst/pubs/feedback-testgen-icse2007.pdf]