Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test gen #186

Merged
merged 52 commits into from Feb 15, 2020
Merged

Test gen #186

merged 52 commits into from Feb 15, 2020

Conversation

@rjust
Copy link
Owner

rjust commented Dec 1, 2018

This PR adds a generic gen_tests script, which calls any of the supported test generators through a simple interface: it exports a set of variables and calls a wrapper shell script which invokes the test generator.

This PR includes wrapper scripts for evosuite and randoop and a test script, which Travis executes in the first testing stage.

This PR also adds a new parameter for test generation, test mode: regression or error-revealing.

@rjust rjust requested review from mernst and jose Dec 1, 2018
framework/bin/gen_tests.pl Outdated Show resolved Hide resolved
framework/bin/gen_tests.pl Outdated Show resolved Hide resolved
framework/bin/gen_tests.pl Outdated Show resolved Hide resolved
framework/bin/gen_tests.pl Outdated Show resolved Hide resolved
@rjust rjust requested a review from Greg4cr Dec 11, 2018
@rjust rjust mentioned this pull request Feb 6, 2019
@Greg4cr

This comment has been minimized.

Copy link
Collaborator

Greg4cr commented Jul 12, 2019

I agree with @jose's comments and I would suggest a four-file system for all test generators:

  • .sh: Wrapper script needed to call the file
  • .credit: Credit file
  • .config: Properties/Command-line parameters.
  • .default.config: Default command-line parameters.

Both EvoSuite and Randoop has several command-line parameters that may be altered for test generation. This should be handled in a way that does not require changing the core script. I definitely run experiments where I feed in all sorts of different EvoSuite configurations, and it would be risky to touch the "evosuite.sh" file with automated scripts.

Instead, scripts could edit the .config file. The .default.config file is there to retain the "default" values, and can be used to restore .config back to "normal".

@Greg4cr

This comment has been minimized.

Copy link
Collaborator

Greg4cr commented Jul 12, 2019

Addressing the core questions in your opening post:
Are the default configurations in the wrapper scripts suitable?

They seem fine, but I would also put them in a separate config file (see above)

Are the "credit" files correct?

They are the papers I would choose. You can, of course, contact the authors and ask what they would prefer.

Should this PR immediately delete the obsolete scripts run_evosuite.pl and run_randoop.pl?

Yes. Some experiment scripting (including my own) will need to be updated, but I think it would be confusing to have multiple ways to run test generation in the framework.

Should this PR immediately delete the randoop/evosuite functionality from Project.pm?

See above. Let's enable test generation in a consistent, universal manner.

@mernst
mernst approved these changes Dec 28, 2019
@rjust rjust requested review from mernst and jose Feb 9, 2020
@rjust

This comment has been minimized.

Copy link
Owner Author

rjust commented Feb 9, 2020

Hi @Greg4cr, @jose, @mernst,

I addressed all the comments. Can you please take another look and let me know whether this is ready for merge?

Note that I also added support for error-revealing vs. regression test generation.

Thanks,
René

@Greg4cr

This comment has been minimized.

Copy link
Collaborator

Greg4cr commented Feb 10, 2020

I have added a script that can test the test generation process for one or more specified bugs: framework/test/test_generate_suites.sh. I think this branch is ready to merge into master once we decide on which we want to do of the following:

  1. Keep test_generate_suites.sh and test_test_gen.sh seperate.
  2. Replace test_test_gen.sh with test_generate_suites.sh (I don't care what filename we use).
@rjust

This comment has been minimized.

Copy link
Owner Author

rjust commented Feb 15, 2020

Thanks for providing the script, Greg!

To avoid confusion, I removed the old test_gen_tests.sh script and updated the travis config.

Copy link
Collaborator

Greg4cr left a comment

Looks good to me.

@rjust rjust merged commit d9952c2 into master Feb 15, 2020
0 of 2 checks passed
0 of 2 checks passed
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details
@rjust rjust deleted the test_gen branch Feb 15, 2020
rjust added a commit that referenced this pull request Feb 15, 2020
* Added a generic test-generation script.
* Added wrapper scripts for Randoop and EvoSuite.
* Made test mode (regression vs. error-revealing) a parameter.
* Export list of relevant (loaded) classes to d4j properties file.
* Added script to test test generation for arbitrary bugs
* Removed test-gen functionality from Project module.

Co-authored-by: Michael Ernst <mernst@alum.mit.edu>
Co-authored-by: Gregory Gay <greg@greggay.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.