No (or very little) test generated for some Apache projects

[jiseongg]

Hi, I'm Jiseong, and have an issue with running Randoop on some large Apache projects. The situation is simple. No test, actually very little test is generated. It seems that many operations are not functioning well for my benchmarks, considering the steps are increasing fast in the progress log.

I've searched some keyword similar to "No tests generated", but I failed to find the same situation in the Issues and Troubleshooting section. If there's an answer somewhere, sorry to bother :(

Environment

• Randoop version: tags/v4.3.0, tags/v4.3.1 (built with Java 11)
• Java version: 11

Picked up JAVA_TOOL_OPTIONS: -Dapple.awt.UIElement=true
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

• Operating System : MacOS Monterey 12.4

Reproduction

• Subject Program: Apache Qpid Proton-j
• CUT : EncoderImpl.java

# prepare the program
git clone https://github.com/apache/qpid-proton-j
cd qpid-proton-j

# build with maven 3.8.5 with Java 11 (JAVA_HOME must be set)
mvn compile package dependency:copy-dependencies -DskipTests -DskipITs -DfailIfNoTests=false

# Testing a class in the module `proton-j`
cd proton-j
# Classpath includes every .jar in target/dependency, target/classes, and the jar of randoop
java -classpath $(printf %s: target/dependency/*.jar)target/classes:/Users/jiseong/github/randoop/randoop/build/libs/randoop-all-4.3.1.jar \
  randoop.main.Main gentests --testclass=org.apache.qpid.proton.codec.EncoderImpl \
  --junit-package-name=org.apache.qpid.proton.codec \
  --testsperfile=1000000 --time-limit=120 --operation-history-log=operation-log.txt

Issues

• Randoop output

// org/apache/qpid/proton/codec/RegressionTest0.java
package org.apache.qpid.proton.codec;

import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class RegressionTest0 {

  public static boolean debug = false;

  @Test
  public void test1() throws Throwable {
    if (debug)
      System.out.format("%n%s%n", "RegressionTest0.test1");
    org.apache.qpid.proton.codec.DecoderImpl decoderImpl0 = null;
    // The following exception was thrown during execution in test generation
    try {
      org.apache.qpid.proton.codec.EncoderImpl encoderImpl1 = new org.apache.qpid.proton.codec.EncoderImpl(decoderImpl0);
      org.junit.Assert.fail("Expected exception of type java.lang.NullPointerException; message: Cannot invoke \"org.apache.qpid.proton.codec.DecoderImpl.register(org.apache.qpid.proton.codec.PrimitiveType)\" because \"decoder\" is null");
    } catch (java.lang.NullPointerException e) {
      // Expected exception.
    }
  }

  @Test
  public void test2() throws Throwable {
    if (debug)
      System.out.format("%n%s%n", "RegressionTest0.test2");
    java.lang.Object obj0 = new java.lang.Object();
    java.lang.Class<?> wildcardClass1 = obj0.getClass();
    org.junit.Assert.assertNotNull(wildcardClass1);
  }
}

Minor question: It seems that test2 is not for testing my CUT. Why does Randoop generate this one?

• Sample output expected

// The shape of test I expected
@Test
public void test() {
  DecoderImpl decoderImpl0 = new DecoderImpl();
  EncoderImpl encoderImpl0 = new EncoderImpl(decoderImpl0);
  Map<String, Object> map0 = new HashMap<>();
  map0.put("a", new Object());
  encoderImpl0.writeMap(map0);
}

// The sample tests that other test generation tools generated
// 1. EvoSuite
@Test
public void test12()  throws Throwable  {
  DecoderImpl decoderImpl0 = new DecoderImpl();
  EncoderImpl encoderImpl0 = new EncoderImpl(decoderImpl0);
  long[] longArray0 = new long[2];
  // Undeclared exception!
  encoderImpl0.writeArray(longArray0);
}

// 2. Research artifact (WIP privately) in my lab
@Test
public void test() {
  DecoderImpl decoderImpl2 = new DecoderImpl();
  encoderImpl2 = new EncoderImpl(decoderImpl2);
  boolean[] booleanArray0 = new boolean[]{ true, true, true };
  encoderImpl2.writeArray(booleanArray0);
}

Misc. Log files

• terminal output

Picked up JAVA_TOOL_OPTIONS: -Dapple.awt.UIElement=true
Randoop for Java version "4.3.1, local changes, branch HEAD, commit a064326, 2022-07-14".

Will try to generate tests for 1 classes.
PUBLIC MEMBERS=64
Explorer = ForwardGenerator(steps: 0, null steps: 0, num_sequences_generated: 0;
    allSequences: 0, regresson seqs: 0, error seqs: 0=0=0, invalid seqs: 0, subsumed_sequences: 0, num_failed_output_test: 0;
    sideEffectFreeMethods: 1113, runtimePrimitivesSeen: 38)

Progress update: steps=1, test inputs generated=0, failing inputs=0      (2022-07-19T23:09:34.832539Z     55.5M used)
Progress update: steps=1000, test inputs generated=3, failing inputs=0      (2022-07-19T23:09:35.435366Z     44.3M used)
...
Progress update: steps=31066000, test inputs generated=3, failing inputs=0      (2022-07-19T23:11:34.824553Z     157M used)
Progress update: steps=31067000, test inputs generated=3, failing inputs=0      (2022-07-19T23:11:34.828322Z     157M used)
Progress update: steps=31067708, test inputs generated=3, failing inputs=0      (2022-07-19T23:11:34.831043Z     165M used)
Normal method executions: 20
Exceptional method executions: 1

Average method execution time (normal termination):      0.00371
Average method execution time (exceptional termination): 1.27
Approximate memory usage 165M
Explorer = ForwardGenerator(steps: 31067708, null steps: 31067705, num_sequences_generated: 3;
    allSequences: 3, regresson seqs: 2, error seqs: 0=0=0, invalid seqs: 0, subsumed_sequences: 0, num_failed_output_test: 1;
    sideEffectFreeMethods: 1113, runtimePrimitivesSeen: 38)

No error-revealing tests to output.

About to look for failing assertions in 2 regression sequences.

Regression test output:
Regression test count: 2
Writing regression JUnit tests...

                                                                                                                                                                                                       
Created file /Users/jiseong/github/apache/qpid-proton-j/proton-j/org/apache/qpid/proton/codec/RegressionTest0.java
Created file /Users/jiseong/github/apache/qpid-proton-j/proton-j/org/apache/qpid/proton/codec/RegressionTest.java
Wrote regression JUnit tests.
About to look for flaky methods.

Invalid tests generated: 0

• --operation-history-log

Operation History:
Operation                                                                                                                                           	ERROR_SEQUENCE	REGRESSION_SEQUENCE	NO_INPUTS_FOUND	REMOVED	SUBSUMED	SEQUENCE_DISCARDED
java.lang.Object.<init>()                                                                                                                           	             0	                  0	              0	      1	       0	                 0
java.lang.Object.getClass(java.lang.Object)                                                                                                         	             0	                  1	              2	      0	       0	            491832
org.apache.qpid.proton.codec.EncoderImpl.<init>(org.apache.qpid.proton.codec.DecoderImpl)                                                           	             0	                  1	              0	      0	       0	            493216
org.apache.qpid.proton.codec.EncoderImpl.getBuffer(org.apache.qpid.proton.codec.EncoderImpl)                                                        	             0	                  0	         492594	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.getDecoder(org.apache.qpid.proton.codec.EncoderImpl)                                                       	             0	                  0	         492794	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.getNullTypeEncoder(org.apache.qpid.proton.codec.EncoderImpl)                                               	             0	                  0	         493124	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.getType(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Object)                                        	             0	                  0	         492919	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.getTypeFromClass(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Class)                                	             0	                  0	         493931	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.registerDescribedType(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Class, java.lang.Object)         	             0	                  0	         493052	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.setByteBuffer(org.apache.qpid.proton.codec.EncoderImpl, java.nio.ByteBuffer)                               	             0	                  0	         491883	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.setByteBuffer(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.codec.WritableBuffer)       	             0	                  0	         493408	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, boolean[])                                            	             0	                  0	         493266	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, byte[])                                               	             0	                  0	         492393	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, char[])                                               	             0	                  0	         493376	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, double[])                                             	             0	                  0	         492977	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, float[])                                              	             0	                  0	         493327	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, int[])                                                	             0	                  0	         493860	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Object[])                                   	             0	                  0	         492214	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, long[])                                               	             0	                  0	         494166	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeArray(org.apache.qpid.proton.codec.EncoderImpl, short[])                                              	             0	                  0	         494090	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeBinary(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.Binary)                  	             0	                  0	         494154	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeBoolean(org.apache.qpid.proton.codec.EncoderImpl, boolean)                                            	             0	                  0	         493000	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeBoolean(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Boolean)                                  	             0	                  0	         492420	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeByte(org.apache.qpid.proton.codec.EncoderImpl, byte)                                                  	             0	                  0	         493449	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeByte(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Byte)                                        	             0	                  0	         492520	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeCharacter(org.apache.qpid.proton.codec.EncoderImpl, char)                                             	             0	                  0	         493791	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeCharacter(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Character)                              	             0	                  0	         493414	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeDecimal128(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.Decimal128)          	             0	                  0	         493418	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeDecimal32(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.Decimal32)            	             0	                  0	         493034	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeDecimal64(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.Decimal64)            	             0	                  0	         491847	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeDescribedType(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.DescribedType)    	             0	                  0	         493258	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeDouble(org.apache.qpid.proton.codec.EncoderImpl, double)                                              	             0	                  0	         493973	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeDouble(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Double)                                    	             0	                  0	         494049	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeFloat(org.apache.qpid.proton.codec.EncoderImpl, float)                                                	             0	                  0	         492992	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeFloat(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Float)                                      	             0	                  0	         492374	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeInteger(org.apache.qpid.proton.codec.EncoderImpl, int)                                                	             0	                  0	         493170	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeInteger(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Integer)                                  	             0	                  0	         493935	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeList(org.apache.qpid.proton.codec.EncoderImpl, java.util.List)                                        	             0	                  0	         492782	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeLong(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Long)                                        	             0	                  0	         491746	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeLong(org.apache.qpid.proton.codec.EncoderImpl, long)                                                  	             0	                  0	         493502	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeMap(org.apache.qpid.proton.codec.EncoderImpl, java.util.Map)                                          	             0	                  0	         493763	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeNull(org.apache.qpid.proton.codec.EncoderImpl)                                                        	             0	                  0	         493071	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeObject(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Object)                                    	             0	                  0	         493536	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, byte)                                                   	             0	                  0	         493990	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, double)                                                 	             0	                  0	         493638	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, float)                                                  	             0	                  0	         493171	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, int)                                                    	             0	                  0	         493650	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, java.lang.String)                                       	             0	                  0	         493338	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, long)                                                   	             0	                  0	         493089	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, short)                                                  	             0	                  0	         493383	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeRaw(org.apache.qpid.proton.codec.EncoderImpl, byte[], int, int)                                       	             0	                  0	         493352	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeShort(org.apache.qpid.proton.codec.EncoderImpl, java.lang.Short)                                      	             0	                  0	         491756	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeShort(org.apache.qpid.proton.codec.EncoderImpl, short)                                                	             0	                  0	         493822	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeString(org.apache.qpid.proton.codec.EncoderImpl, java.lang.String)                                    	             0	                  0	         494012	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeSymbol(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.Symbol)                  	             0	                  0	         493067	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeTimestamp(org.apache.qpid.proton.codec.EncoderImpl, java.util.Date)                                   	             0	                  0	         492651	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeTimestamp(org.apache.qpid.proton.codec.EncoderImpl, long)                                             	             0	                  0	         492071	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeUUID(org.apache.qpid.proton.codec.EncoderImpl, java.util.UUID)                                        	             0	                  0	         493635	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeUnsignedByte(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.UnsignedByte)      	             0	                  0	         494153	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeUnsignedInteger(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.UnsignedInteger)	             0	                  0	         492503	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeUnsignedLong(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.UnsignedLong)      	             0	                  0	         492585	      0	       0	                 0
org.apache.qpid.proton.codec.EncoderImpl.writeUnsignedShort(org.apache.qpid.proton.codec.EncoderImpl, org.apache.qpid.proton.amqp.UnsignedShort)    	             0	                  0	         491778	      0	       0	                 0

Since I'm planning to compare my work with Randoop, I want to know whether I've misused it or not. Thanks in advance for any help you're able to provide.