Test gen

[rjust]

This PR adds a generic gen_tests script, which calls any of the supported test generators through a simple interface: it exports a set of variables and calls a wrapper shell script which invokes the test generator.

This PR includes wrapper scripts for evosuite and randoop and a test script, which Travis executes in the first testing stage.

This PR also adds a new parameter for test generation, test mode: regression or error-revealing.

[Greg4cr]

I agree with @jose's comments and I would suggest a four-file system for all test generators:

• .sh: Wrapper script needed to call the file
• .credit: Credit file
• .config: Properties/Command-line parameters.
• .default.config: Default command-line parameters.

Both EvoSuite and Randoop has several command-line parameters that may be altered for test generation. This should be handled in a way that does not require changing the core script. I definitely run experiments where I feed in all sorts of different EvoSuite configurations, and it would be risky to touch the "evosuite.sh" file with automated scripts.

Instead, scripts could edit the .config file. The .default.config file is there to retain the "default" values, and can be used to restore .config back to "normal".

[Greg4cr]

Addressing the core questions in your opening post:
Are the default configurations in the wrapper scripts suitable?

They seem fine, but I would also put them in a separate config file (see above)

Are the "credit" files correct?

They are the papers I would choose. You can, of course, contact the authors and ask what they would prefer.

Should this PR immediately delete the obsolete scripts run_evosuite.pl and run_randoop.pl?

Yes. Some experiment scripting (including my own) will need to be updated, but I think it would be confusing to have multiple ways to run test generation in the framework.

Should this PR immediately delete the randoop/evosuite functionality from Project.pm?

See above. Let's enable test generation in a consistent, universal manner.

[rjust]

Hi @Greg4cr, @jose, @mernst,

I addressed all the comments. Can you please take another look and let me know whether this is ready for merge?

Note that I also added support for error-revealing vs. regression test generation.

Thanks,
René

[Greg4cr]

I have added a script that can test the test generation process for one or more specified bugs: framework/test/test_generate_suites.sh. I think this branch is ready to merge into master once we decide on which we want to do of the following:

1. Keep test_generate_suites.sh and test_test_gen.sh seperate.
2. Replace test_test_gen.sh with test_generate_suites.sh (I don't care what filename we use).

[rjust]

Thanks for providing the script, Greg!

To avoid confusion, I removed the old test_gen_tests.sh script and updated the travis config.

[Greg4cr]

Looks good to me.