Load test results into DB by wtwhite · Pull Request #43 · binaryeq/differential-test-generation

wtwhite · 2024-11-14T23:31:52Z

The JaCoCo coverage results are already in the DB; in order to figure out which coverage differences do not correspond to differences in test results, we need to get the latter into the DB too.

…m generated tests between pairs of providers

…iles into the DB

wtwhite · 2024-11-14T23:36:27Z

There are some discrepancies between the test classes in the test_classes table (built from test_results_{obfs,gaoss,mvnc}.tsv, which are extracted from raw JUnit4 runner outputs) and in the tests table (gathered from JaCoCo XML reports):

sqlite> select count(*) from (select provider, groupId, artifactId, version, testClass from test_classes EXCEPT select * from tests);
┌──────────┐
│ count(*) │
├──────────┤
│ 40       │
└──────────┘
sqlite> select count(*) from (select * from tests EXCEPT select provider, groupId, artifactId, version, testClass from test_classes);
┌──────────┐
│ count(*) │
├──────────┤
│ 19       │
└──────────┘
sqlite> select * from tests EXCEPT select provider, groupId, artifactId, version, testClass from test_classes;
┌──────────┬───────────────────────┬───────────────────────┬──────────────┬───────────────────────────────────────────────────────────────────────────────┐
│ provider │        groupId        │      artifactId       │   version    │                                     test                                      │
├──────────┼───────────────────────┼───────────────────────┼──────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ gaoss    │ com/google/jimfs      │ jimfs                 │ 1.2          │ com.google.common.jimfs.PathURLConnection_ESTest                              │
│ gaoss    │ commons-configuration │ commons-configuration │ 1.10         │ org.apache.commons.configuration.AbstractHierarchicalFileConfiguration_ESTest │
│ gaoss    │ commons-logging       │ commons-logging       │ 1.2          │ org.apache.commons.logging.impl.LogFactoryImpl_ESTest                         │
│ gaoss    │ io/rest-assured       │ rest-assured          │ 5.2.1        │ io.restassured.internal.matcher.xml.XmlXsdMatcher_ESTest                      │
│ gaoss    │ org/apache/avro       │ avro                  │ 1.11.1       │ org.apache.avro.file.DataFileWriter_ESTest                                    │
│ mvnc     │ com/google/jimfs      │ jimfs                 │ 1.2          │ com.google.common.jimfs.PathURLConnection_ESTest                              │
│ mvnc     │ commons-configuration │ commons-configuration │ 1.10         │ org.apache.commons.configuration.AbstractHierarchicalFileConfiguration_ESTest │
│ mvnc     │ commons-logging       │ commons-logging       │ 1.2          │ org.apache.commons.logging.impl.LogFactoryImpl_ESTest                         │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.channel.socket.nio.NioServerBoss_ESTest                       │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink_ESTest         │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.handler.codec.frame.FrameDecoder_ESTest                       │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.handler.codec.marshalling.CompatibleMarshallingDecoder_ESTest │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.handler.codec.marshalling.MarshallingDecoder_ESTest           │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.handler.ssl.OpenSslEngine_ESTest                              │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.util.ThreadRenamingRunnable_ESTest                            │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.util.Version_ESTest                                           │
│ mvnc     │ io/netty              │ netty                 │ 3.10.6.Final │ org.jboss.netty.util.internal.NativeLibraryLoader_ESTest                      │
│ mvnc     │ io/rest-assured       │ rest-assured          │ 5.2.1        │ io.restassured.internal.matcher.xml.XmlXsdMatcher_ESTest                      │
│ mvnc     │ org/apache/avro       │ avro                  │ 1.11.1       │ org.apache.avro.file.DataFileWriter_ESTest                                    │
└──────────┴───────────────────────┴───────────────────────┴──────────────┴───────────────────────────────────────────────────────────────────────────────┘

wtwhite · 2024-11-14T23:42:02Z

Indeed there are 19 JUnit4 reports that failed parsing:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ find run -name '*.out.tsv.failed'|wc -l
19

Looking at the example of com.google.common.jimfs.PathURLConnection_ESTest, noise has corrupted the output, but it should still be possible to extract everything we want:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ cat !$
cat run/gaoss/com/google/jimfs/jimfs/1.2/com.google.common.jimfs.PathURLConnection_ESTest.out
JUnit version 4.13.1
..........E.............
Time: 1.628
There was 1 failure:
1) test21(com.google.common.jimfs.PathURLConnection_ESTest)
java.lang.ExceptionInInitializerError
	at java.nio.file.Files.probeContentType(Files.java:1617)
	at com.google.common.jimfs.PathURLConnection.connect(PathURLConnection.java:97)
	at com.google.common.jimfs.PathURLConnection.getInputStream(PathURLConnection.java:121)
	at com.google.common.jimfs.PathURLConnection_ESTest.test21(PathURLConnection_ESTest.java:365)
	... 11 trimmed
Caused by: java.lang.SecurityException: Security manager blocks ("java.lang.RuntimePermission" "fileTypeDetector")
java.lang.Thread.getStackTrace(Thread.java:1564)
org.evosuite.runtime.sandbox.MSecurityManager.checkPermission(MSecurityManager.java:424)
java.nio.file.spi.FileTypeDetector.checkPermission(FileTypeDetector.java:55)
java.nio.file.spi.FileTypeDetector.<init>(FileTypeDetector.java:68)
sun.nio.fs.AbstractFileTypeDetector.<init>(AbstractFileTypeDetector.java:41)
sun.nio.fs.GnomeFileTypeDetector.<init>(GnomeFileTypeDetector.java:50)
sun.nio.fs.LinuxFileSystemProvider.getFileTypeDetector(LinuxFileSystemProvider.java:109)
sun.nio.fs.DefaultFileTypeDetector.create(DefaultFileTypeDetector.java:37)
java.nio.file.Files$FileTypeDetectors$1.run(Files.java:1546)
java.nio.file.Files$FileTypeDetectors$1.run(Files.java:1544)
java.security.AccessController.doPrivileged(Native Method)
java.nio.file.Files$FileTypeDetectors.createDefaultFileTypeDetector(Files.java:1544)
java.nio.file.Files$FileTypeDetectors.<clinit>(Files.java:1537)
java.nio.file.Files.probeContentType(Files.java:1617)
com.google.common.jimfs.PathURLConnection.connect(PathURLConnection.java:97)
com.google.common.jimfs.PathURLConnection.getInputStream(PathURLConnection.java:121)
com.google.common.jimfs.PathURLConnection_ESTest.test21(PathURLConnection_ESTest.java:365)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.lang.Thread.run(Thread.java:750)

	at org.evosuite.runtime.sandbox.MSecurityManager.checkPermission(MSecurityManager.java:436)
	at java.nio.file.spi.FileTypeDetector.checkPermission(FileTypeDetector.java:55)
	at java.nio.file.spi.FileTypeDetector.<init>(FileTypeDetector.java:68)
	at sun.nio.fs.AbstractFileTypeDetector.<init>(AbstractFileTypeDetector.java:41)
	at sun.nio.fs.GnomeFileTypeDetector.<init>(GnomeFileTypeDetector.java:50)
	at sun.nio.fs.LinuxFileSystemProvider.getFileTypeDetector(LinuxFileSystemProvider.java:109)
	at sun.nio.fs.DefaultFileTypeDetector.create(DefaultFileTypeDetector.java:37)
	at java.nio.file.Files$FileTypeDetectors$1.run(Files.java:1546)
	at java.nio.file.Files$FileTypeDetectors$1.run(Files.java:1544)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.nio.file.Files$FileTypeDetectors.createDefaultFileTypeDetector(Files.java:1544)
	at java.nio.file.Files$FileTypeDetectors.<clinit>(Files.java:1537)
	... 16 more

FAILURES!!!
Tests run: 23,  Failures: 1

Next steps:

Eyeball all 19; see if I can relax the parser to get them all or if some are beyond saving.
These results are also missing from the spreadsheet -- update it.

wtwhite · 2024-11-15T01:02:33Z

There are now no tests that JaCoCo picks up but not our test result parsing:

sqlite> select count(*) from (select * from tests EXCEPT select provider, groupId, artifactId, version, testClass from test_classes);
┌──────────┐
│ count(*) │
├──────────┤
│ 0        │
└──────────┘

Though there are still the same 40 tests missing in the other direction:

sqlite> select count(*) from (select provider, groupId, artifactId, version, testClass from test_classes EXCEPT select * from tests);
┌──────────┐
│ count(*) │
├──────────┤
│ 40       │
└──────────┘
sqlite> select provider, groupId, artifactId, version, testClass from test_classes EXCEPT select * from tests;
┌──────────┬──────────────────────────┬──────────────────┬───────────────┬─────────────────────────────────────────────────────────────────────────┐
│ provider │         groupId          │    artifactId    │    version    │                                testClass                                │
├──────────┼──────────────────────────┼──────────────────┼───────────────┼─────────────────────────────────────────────────────────────────────────┤
│ gaoss    │ com/google/auto          │ auto-common      │ 1.2.1         │ com.google.auto.common.BasicAnnotationProcessor_ESTest                  │
│ gaoss    │ com/google/auto          │ auto-common      │ 1.2.2         │ com.google.auto.common.BasicAnnotationProcessor_ESTest                  │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.collect.MapMakerInternalMap_ESTest                    │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.io.ByteSink_ESTest                                    │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.math.ToDoubleRounder_ESTest                           │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.reflect.TypeVisitor_ESTest                            │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractExecutionThreadService_ESTest │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractIdleService_ESTest            │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractService_ESTest                │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.InterruptibleTask_ESTest              │
│ gaoss    │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.WrappingExecutorService_ESTest        │
│ gaoss    │ com/google/truth         │ truth            │ 1.1.3         │ com.google.common.truth.AutoValue_ActualValueInference_FrameInfo_ESTest │
│ gaoss    │ com/google/truth         │ truth            │ 1.1.3         │ com.google.common.truth.ComparisonFailures_ESTest                       │
│ gaoss    │ com/google/truth         │ truth            │ 1.1.3         │ com.google.common.truth.MultimapSubject_ESTest                          │
│ gaoss    │ io/netty                 │ netty-codec-http │ 4.1.102.Final │ io.netty.handler.codec.http.LastHttpContent_ESTest                      │
│ gaoss    │ io/netty                 │ netty-codec-http │ 4.1.102.Final │ io.netty.handler.codec.spdy.SpdyHeaders_ESTest                          │
│ gaoss    │ org/apache/logging/log4j │ log4j-api        │ 2.17.1        │ org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest           │
│ gaoss    │ org/threeten             │ threetenbp       │ 1.6.6         │ org.threeten.bp.temporal.IsoFields_ESTest                               │
│ gaoss    │ org/threeten             │ threetenbp       │ 1.6.7         │ org.threeten.bp.temporal.IsoFields_ESTest                               │
│ mvnc     │ com/google/auto          │ auto-common      │ 1.2.1         │ com.google.auto.common.BasicAnnotationProcessor_ESTest                  │
│ mvnc     │ com/google/auto          │ auto-common      │ 1.2.2         │ com.google.auto.common.BasicAnnotationProcessor_ESTest                  │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.collect.MapMakerInternalMap_ESTest                    │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.io.ByteSink_ESTest                                    │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.math.ToDoubleRounder_ESTest                           │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.reflect.TypeVisitor_ESTest                            │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractExecutionThreadService_ESTest │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractIdleService_ESTest            │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractService_ESTest                │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.InterruptibleTask_ESTest              │
│ mvnc     │ com/google/guava         │ guava            │ 31.0.1-jre    │ com.google.common.util.concurrent.WrappingExecutorService_ESTest        │
│ mvnc     │ com/google/guava         │ guava-testlib    │ 32.1.2-jre    │ com.google.common.collect.testing.AbstractIteratorTester_ESTest         │
│ mvnc     │ com/google/truth         │ truth            │ 1.1.3         │ com.google.common.truth.AutoValue_ActualValueInference_FrameInfo_ESTest │
│ mvnc     │ com/google/truth         │ truth            │ 1.1.3         │ com.google.common.truth.ComparisonFailures_ESTest                       │
│ mvnc     │ com/google/truth         │ truth            │ 1.1.3         │ com.google.common.truth.MultimapSubject_ESTest                          │
│ mvnc     │ io/netty                 │ netty-codec-http │ 4.1.102.Final │ io.netty.handler.codec.http.LastHttpContent_ESTest                      │
│ mvnc     │ io/netty                 │ netty-codec-http │ 4.1.102.Final │ io.netty.handler.codec.spdy.SpdyHeaders_ESTest                          │
│ mvnc     │ org/apache/logging/log4j │ log4j-api        │ 2.17.1        │ org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest           │
│ mvnc     │ org/threeten             │ threetenbp       │ 1.6.6         │ org.threeten.bp.temporal.IsoFields_ESTest                               │
│ mvnc     │ org/threeten             │ threetenbp       │ 1.6.7         │ org.threeten.bp.temporal.IsoFields_ESTest                               │
│ obfs     │ com/google/guava         │ guava-testlib    │ 32.1.2-jre    │ com.google.common.collect.testing.AbstractIteratorTester_ESTest         │
└──────────┴──────────────────────────┴──────────────────┴───────────────┴─────────────────────────────────────────────────────────────────────────┘

wtwhite · 2024-11-15T01:10:15Z

Looking at the first one, com.google.auto.common.BasicAnnotationProcessor_ESTest, EvoSuite actually only generated a single "test" method for it that does nothing:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ cat testgen/mvnc/com/google/auto/auto-common/1.2.1/evosuite-tests/com/google/auto/common/BasicAnnotationProcessor_ESTest.java
--snip--
@RunWith(EvoRunner.class) @EvoRunnerParameters(mockJVMNonDeterminism = true, useVFS = true, useVNET = true, resetStaticState = true, separateClassLoader = true) 
public class BasicAnnotationProcessor_ESTest extends BasicAnnotationProcessor_ESTest_scaffolding {
  @Test
  public void notGeneratedAnyTest() {
    // EvoSuite did not generate any tests
  }
}

The test succeeds, unsurprisingly:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ cat run/gaoss/com/google/auto/auto-common/1.2.1/com.google.auto.common.BasicAnnotationProcessor_ESTest.out
JUnit version 4.13.2
.
Time: 0.002

OK (1 test)

But no .jacoco.exec file is created when the test is run, probably because the JaCoCo-instrumented class does not even get loaded in the first place.

wtwhite · 2024-11-15T01:26:13Z

In fact, 38 of the 40 are in the same situation. I gathered all relevant generated test source names with select 'testgen/mvnc/' || groupId || '/' || artifactId || '/' || version || '/evosuite-tests/' || replace(testClass, '.', '/') || '.java' from (select provider, groupId, artifactId, version, testClass from test_classes EXCEPT select * from tests); into generated_tests_that_jacoco_misses.txt and then:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ grep --files-without-match '// EvoSuite did not generate any tests' `cat generated_tests_that_jacoco_misses.txt`
testgen/mvnc/org/apache/logging/log4j/log4j-api/2.17.1/evosuite-tests/org/apache/logging/log4j/spi/DefaultThreadContextStack_ESTest.java
testgen/mvnc/org/apache/logging/log4j/log4j-api/2.17.1/evosuite-tests/org/apache/logging/log4j/spi/DefaultThreadContextStack_ESTest.java

Interestingly it has generated a .jacoco.exec file, but no corresponding XML report:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ lll run/gaoss/org/apache/logging/log4j/log4j-api/2.17.1
total 24
drwxrwxr-x 3 wtwhite wtwhite 4096 Nov 14 03:51 ..
-rw-rw-r-- 1 wtwhite wtwhite  741 Nov 14 11:21 org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.out
-rw-rw-r-- 1 wtwhite wtwhite 2684 Nov 14 11:21 org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.jacoco.exec
-rw-rw-r-- 1 wtwhite wtwhite  385 Nov 14 11:21 org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.err
drwxrwxr-x 2 wtwhite wtwhite 4096 Nov 15 13:31 .
-rw-rw-r-- 1 wtwhite wtwhite  135 Nov 15 13:57 org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.out.tsv

Its test output is a failure, but it looks "normal":

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ cat run/gaoss/org/apache/logging/log4j/log4j-api/2.17.1/org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.out
JUnit version 4.13.2
............................................E......
Time: 0.625
There was 1 failure:
1) test20(org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest)
java.lang.AssertionError: Exception was not thrown in org.apache.logging.log4j.spi.MutableThreadContextStack but in org.apache.logging.log4j.spi.DefaultThreadContextStack.formatTo(DefaultThreadContextStack.java:289): java.lang.NullPointerException
	at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
	at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
	at org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.test20(DefaultThreadContextStack_ESTest.java:253)

FAILURES!!!
Tests run: 50,  Failures: 1

The underlying issue is that jacococli report crashes while trying to make the XML report. Repro:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ java -jar /home/wtwhite/code/regression-test-generation/find-diff-classes/jacoco/lib/jacococli.jar report "/home/wtwhite/code/regression-test-generation/find-diff-classes/run/gaoss/org/apache/logging/log4j/log4j-api/2.17.1/org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.jacoco.exec" --classfiles "/home/wtwhite/code/craw-redhat-oss/wget/crawl/google-aoss/org/apache/logging/log4j/log4j-api/2.17.1/log4j-api-2.17.1.jar" --xml repro.xml
[INFO] Loading execution data file /home/wtwhite/code/regression-test-generation/find-diff-classes/run/gaoss/org/apache/logging/log4j/log4j-api/2.17.1/org.apache.logging.log4j.spi.DefaultThreadContextStack_ESTest.jacoco.exec.
Exception in thread "main" java.io.IOException: Error while analyzing /home/wtwhite/code/craw-redhat-oss/wget/crawl/google-aoss/org/apache/logging/log4j/log4j-api/2.17.1/log4j-api-2.17.1.jar@org/apache/logging/log4j/util/StackLocator.class with JaCoCo 0.8.12.202403310830/dbfb6f2.
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzerError(Analyzer.java:163)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:135)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:158)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeAll(Analyzer.java:195)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeZip(Analyzer.java:267)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeAll(Analyzer.java:198)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeAll(Analyzer.java:228)
	at org.jacoco.cli.internal.commands.Report.analyze(Report.java:110)
	at org.jacoco.cli.internal.commands.Report.execute(Report.java:84)
	at org.jacoco.cli.internal.Main.execute(Main.java:90)
	at org.jacoco.cli.internal.Main.main(Main.java:105)
Caused by: java.lang.IllegalStateException: Can't add different class with same name: org/apache/logging/log4j/util/StackLocator
	at org.jacoco.cli.internal.core.analysis.CoverageBuilder.visitCoverage(CoverageBuilder.java:106)
	at org.jacoco.cli.internal.core.analysis.Analyzer$1.visitEnd(Analyzer.java:100)
	at org.jacoco.cli.internal.asm.ClassVisitor.visitEnd(ClassVisitor.java:395)
	at org.jacoco.cli.internal.core.internal.flow.ClassProbesAdapter.visitEnd(ClassProbesAdapter.java:100)
	at org.jacoco.cli.internal.asm.ClassReader.accept(ClassReader.java:749)
	at org.jacoco.cli.internal.asm.ClassReader.accept(ClassReader.java:425)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:117)
	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:133)
	... 9 more
wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ lll repro.xml
ls: cannot access 'repro.xml': No such file or directory

wtwhite · 2024-11-15T02:29:01Z

As it's a single test that triggers this jacococli report crash, I'll just ignore it for now. (I found this but it doesn't look trivial to accomplish.)

We can now easily find all 60 individual test methods that fail for one provider but not another:

sqlite> select c1.provider as provider1, c2.provider as provider2, c1.groupId, c1.artifactId, c1.version, c1.testClass, f1.testMethodSuffix from test_classes c1 join test_classes c2 on (c2.groupId, c2.artifactId, c2.version, c2.testClass) = (c1.groupId, c1.artifactId, c1.version, c1.testClass) and c1.provider <> c2.provider join failed_test_methods f1 on (f1.provider, f1.groupId, f1.artifactId, f1.version, f1.testClass) = (c1.provider, c1.groupId, c1.artifactId, c1.version, c1.testClass) where not exists (select 1 from failed_test_methods f2 where (f2.provider, f2.groupId, f2.artifactId, f2.version, f2.testClass, f2.testMethodSuffix) = (c2.provider, f1.groupId, f1.artifactId, f1.version, f1.testClass, f1.testMethodSuffix)) order by 1, 2, 3, 4, 5, 6, 7;
┌───────────┬───────────┬─────────────┬──────────────────┬───────────────┬───────────────────────────────────────────────────────────────────┬──────────────────┐
│ provider1 │ provider2 │   groupId   │    artifactId    │    version    │                             testClass                             │ testMethodSuffix │
├───────────┼───────────┼─────────────┼──────────────────┼───────────────┼───────────────────────────────────────────────────────────────────┼──────────────────┤
│ gaoss     │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.filefilter.NotFileFilter_ESTest             │ 00               │
│ gaoss     │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.filefilter.NotFileFilter_ESTest             │ 12               │
│ gaoss     │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.monitor.FileAlterationObserver_ESTest       │ 04               │
│ gaoss     │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.monitor.FileAlterationObserver_ESTest       │ 17               │
│ gaoss     │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.output.UncheckedAppendableImpl_ESTest       │ 13               │
│ gaoss     │ mvnc      │ io/netty    │ netty-codec-http │ 4.1.102.Final │ io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest │ 50               │
│ gaoss     │ mvnc      │ io/netty    │ netty-codec-http │ 4.1.102.Final │ io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest │ 51               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.23.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 00               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.23.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 01               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.23.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 12               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.23.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 15               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.23.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 16               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 01               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 02               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 03               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 04               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 08               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 09               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 11               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 12               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 16               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 17               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 21               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 22               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 24               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 25               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 26               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 29               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.24.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 32               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.25.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 00               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.25.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 01               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.25.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 16               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.25.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 17               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.25.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 18               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.26.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 00               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.26.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 01               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.26.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 13               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.26.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 17               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.26.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 18               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.26.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 23               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.28.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 00               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.28.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 01               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.28.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 13               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.28.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 15               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.28.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 16               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.31.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 00               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.31.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 01               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.31.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 02               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.31.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 15               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.31.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 16               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.32.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 00               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.32.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 01               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.32.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 02               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.32.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 15               │
│ gaoss     │ mvnc      │ io/undertow │ undertow-servlet │ 2.2.32.Final  │ io.undertow.servlet.spec.UpgradeServletInputStream_ESTest         │ 16               │
│ obfs      │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.filefilter.NotFileFilter_ESTest             │ 00               │
│ obfs      │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.filefilter.NotFileFilter_ESTest             │ 12               │
│ obfs      │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.monitor.FileAlterationObserver_ESTest       │ 04               │
│ obfs      │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.monitor.FileAlterationObserver_ESTest       │ 17               │
│ obfs      │ mvnc      │ commons-io  │ commons-io       │ 2.15.0        │ org.apache.commons.io.output.UncheckedAppendableImpl_ESTest       │ 13               │
└───────────┴───────────┴─────────────┴──────────────────┴───────────────┴───────────────────────────────────────────────────────────────────┴──────────────────┘

…another, another for finding methods that were covered by one provider's tests but not another's

…esults into the DB

…coverage

wtwhite added 2 commits November 15, 2024 11:27

Query to find all 116 methods with different coverage that result fro…

c8185f7

…m generated tests between pairs of providers

Scripts to import tests and individual failed test methods from TSV f…

8e4923d

…iles into the DB

wtwhite self-assigned this Nov 14, 2024

wtwhite added 4 commits November 15, 2024 13:11

Relax parser to successfully pares 13 more test results into TSV

e775256

Further relax parser to get 2 more results. Regex getting crazy

8607b83

Catch 2 more results, just 2 to go

f4ae3da

Finally we get all test results into TSV, no parse failures

64a5818

wtwhite added 5 commits November 15, 2024 15:41

Add 2 views: 1 for finding tests that failed in one provider but not …

d7906a7

…another, another for finding methods that were covered by one provider's tests but not another's

Add clean and really-clean targets to Makefile for importing JaCoCo r…

681eb17

…esults into the DB

Rename column test -> testClass for parity with test_classes

c09abb6

Add jacoco_ prefix to both Jacoco-based DB table names

6b0eb5c

Put the final, interesting query into a view: same_test_results_diff_…

42f0bb5

…coverage

wtwhite merged commit 646cb36 into main Nov 15, 2024

wtwhite deleted the test-results-into-db branch November 15, 2024 03:21

wtwhite mentioned this pull request Nov 15, 2024

66 coverage differences occur despite identical test outcomes #44

Open

wtwhite mentioned this pull request Nov 27, 2024

2 differing test outcomes between mvnc and gaoss in io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest #53

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load test results into DB#43

Load test results into DB#43
wtwhite merged 11 commits into
mainfrom
test-results-into-db

wtwhite commented Nov 14, 2024

Uh oh!

wtwhite commented Nov 14, 2024

Uh oh!

wtwhite commented Nov 14, 2024

Uh oh!

wtwhite commented Nov 15, 2024

Uh oh!

wtwhite commented Nov 15, 2024

Uh oh!

wtwhite commented Nov 15, 2024

Uh oh!

wtwhite commented Nov 15, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

wtwhite commented Nov 14, 2024

Uh oh!

wtwhite commented Nov 14, 2024

Uh oh!

wtwhite commented Nov 14, 2024

Uh oh!

wtwhite commented Nov 15, 2024

Uh oh!

wtwhite commented Nov 15, 2024

Uh oh!

wtwhite commented Nov 15, 2024

Uh oh!

wtwhite commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wtwhite commented Nov 15, 2024 •

edited

Loading