Running Lang 62

[loren138]

This might be related to #25 I'm trying to generate regression tests for Lang 62 in the same style as Defects4J.

If I run:
java -cp /home/loren/pullTest/bertGithub/src/main/java/:/home/loren/pullTest/bertGithub/junit-4.12.jar:/home/loren/pullTest/bertGithub/hamcrest-core-1.3.jar:/home/loren/pullTest/bertGithub/slf4j-nop-1.7.25.jar:/home/loren/ramdisk/Lang_62/version/oldInst/:/home/loren/pullTest/bertGithub/randoop-all-3.1.5.jar randoop.main.Main gentests --testclass=org.apache.commons.lang.Entities --testclass=org.apache.commons.lang.IntHashMap --timelimit=180 --ignore-flaky-tests=true --only-test-public-members=true --clear=10000 --string-maxlen=5000 --forbid-null=false --null-ratio=0.1 --no-error-revealing-tests=true --noprogressdisplay=true --testsperfile=250 --randomseed=3347 --junit-output-dir=/home/loren/ramdisk/Lang_62/version/randoop/

I get

not using package
Ignoring non-visible class org.apache.commons.lang.Entities specified via --classlist or --testclass.
Ignoring non-visible class org.apache.commons.lang.IntHashMap specified via --classlist or --testclass.
No classes to test

This is on Randoop 3.1.5.

It appears that both classes in Apache Lang are indeed package private. (ie declared as just class Name not public class Name) but the Defects4J paper (Do Automatically Generated Unit tests...) shows randoop as running tests on those classes. Do you have any idea if there has been a change in behavior in Randoop since then or was some other setting used for generating those tests?
https://github.com/apache/commons-lang/blob/LANG_2_6/src/main/java/org/apache/commons/lang/Entities.java
https://github.com/apache/commons-lang/blob/LANG_2_6/src/main/java/org/apache/commons/lang/IntHashMap.java

I tried to copy the settings from the perl files, but it's possible I got some of them wrong.

[rjust]

Hi @loren138,

I am not entirely sure how the (default) behavior of Randoop changed, but does Randoop's --junit-package-name option solve this issue for you?

Randoop should be able to test these classes if you provide org.apache.commons.lang as the package name.

Let me know whether this does the trick. If not, I can dig a little deeper into the differences between Randoop v2 and v3.

Thanks,
René

[loren138]

Thanks for the thoughts!

I added in --junit-package-name but it seems to be mutually exclusive to --only-test-public-members which is to true in the in the runner.

defects4j/framework/util/randoop.config

Line 10 in a910322

--only-test-public-members=true

If I leave --only-test-public-members as false (the default) and set --junit-package-name, it will generate tests for those classes.

I think I found it! It looks like only-test-public-members was added to Defects4J 2016 with Randoop 2.1.2. (Paper was from 2015 so without that option.) Do you remember what adding it fixed so I don't break it by removing it?

Related Links:

Randoop Release
Related Commit to Defects4J

[rjust]

If I remember correctly, the recommendation was to set the only-test-public-members to true to reduce the number of flakey tests. It seems, however, that we never tried to regenerate tests for all Defects4J bugs after making this change -- otherwise we would have noticed that Randoop cannot generate tests for some bugs.

Setting the only-test-public-members flag to false seems fine and will certainly give you more tests to work with. I recommend to run the fix_test_suite.pl script after test generation to eliminate flakey tests. This is true regardless of the configuration for test generation.

[loren138]

Unfortunately, in randoop now, you do have to set junit-package-name you can't just set only-test-public-members to false. I think I remember reading that this was a behavior change in randoop.

Randoop only invokes a method or constructor M if all of the following are true:

1. M is declared in a class that is named by --testclass or --classlist, or is a member named by --methodlist.
2. M does not match the pattern given via --omitmethods.
3. All of the following are accessible from the package of the tests (set with --junit-package-name): M's class, M, and M's return type and parameter types.

https://randoop.github.io/randoop/manual/index.html

I guess you'll either need to restrict d4j to a randoop version before that for testing purposes or add package names to the dataset. I've written a little script to grab the package name from the modified classes by just taking the package part of the first one. I don't currently handle if the modified classes are in separate packages.

[rjust]

One possibility is to group target classes by package name (which requires knowledge about the packages) and invoke Randoop once for each distinct package name, using the --junit-package-name option. To avoid redundancy among test cases, each invocation of Randoop could be limited to a set of target classes:
https://randoop.github.io/randoop/manual/index.html#exercised-filter
The number of target classes (i.e., modified classes) is usually very small, and I don't know for sure whether the multiple-packages problem currently exists for any of the 395 defects. Your script and solution may actually be sufficient for now.

Note that you don't have to change Defects4J to play with different configuration options -- you can provide your own randoop.config file by setting the environment variable: RANDOOP_CONFIG_FILE.

Also, we should upgrade the supported version of Randoop in Defects4J to the latest version (3.1.5), so if you have suggestions for better default configuration options that would be great.

Thanks,
René