Example: Generating tests for java.util.Collections resulted flaky test error

[wudc]

Randoop version: 4.0.4

Command ran: D:\tools\Unit_Test\Frameworks\randoop-tutorials\java-collections-test>java -classpath .;%RANDOOP_JAR%\randoop-all-4.0.4.jar randoop.main.Main gentests --classlist=myclasses.txt --time-limit=60

Output:
D:\tools\Unit_Test\Frameworks\randoop-tutorials\java-collections-test>java -classpath .;%RANDOOP_JAR%\randoop-all-4.0.4.jar randoop.main.Main gentests --classlist=myclasses.txt --time-limit=60
PUBLIC MEMBERS=113
Explorer = randoop.generation.ForwardGenerator(allSequences.size()=0,observers.size()=0,subsumed_sequences.size()=0,runtimePrimitivesSeen.size()=38)

Progress update: steps=1, test inputs generated=0, failing inputs=0 (Sat Jun 30 11:48:50 EDT 2018)

ERROR: Randoop stopped because of a flaky test.

This can happen when Randoop is run on methods that side-effect global state.
See the "Randoop stopped because of a flaky test" section of the user manual.
For more details, rerun with logging turned on with --log=FILENAME.

Input file:
myclasses.txt
Log file:
mylog.txt

$(RANDOOP_JAR)=D:\tools\Unit_Test\Frameworks\randoop-4.0.4\jars

[robryb]

Came here to report the same core issue, that the example from the tutorial failed.

HOWEVER, the following commands ran successfully when I expected one (or both) of them to fail:

$ $JAVA_HOME/bin/java -classpath "$JAVA_HOME/bin:$RANDOOP_JAR" randoop.main.Main gentests --testclass=java.util.Collections --time-limit=120

$ $JAVA_HOME/bin/java -classpath "$JAVA_HOME/bin:$RANDOOP_JAR" randoop.main.Main gentests --testclass=java.util.TreeSet --time-limit=120

Collections and TreeSet both come from myclasses.txt. Since "--classlist=myclasses.txt" failed, I therefore expected either "--testclass=java.util.Collections" to fail or "--testclass=java.util.TreeSet" to fail (or both).

Both passed individually but failed when combined in myclasses.txt. That suggests to me that the problem has to to with either the "--classlist=" parameter or with combining these two classes in one "--classlist".

[mernst]

Thanks for reporting the issue.
The underlying problem is that Randoop needs to do a better job modeling Java generic types.
Randoop doesn't catch the problem internally because of a difference between behavior when Java code is executed reflectively versus executed normally.

For now, you can work around the problem by passing the command-line argument --flaky-test-behavior=DISCARD. I have updated the manual so that its example suggests this command-line argument.
Could you let me know whether that resolves your problem?