testCompareDocuments fails during "mvn clean install"

[rnjudge]

Tern builds the spdx tool from source as a part of its CI tests to make sure Tern continues to produce valid SPDX documents. The mvn clean install command is failing as a part of this process. I can reproduce the error in my local environment:

$ git clone https://github.com/spdx/tools-java.git && cd tools-java
$ export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::")
$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64
$ mvn clean install
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.spdx.tools.CompareSpdxDocsTest
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.097 sec <<< FAILURE!
testCompareDocuments(org.spdx.tools.CompareSpdxDocsTest)  Time elapsed: 16.043 sec  <<< FAILURE!
junit.framework.AssertionFailedError
	at junit.framework.Assert.fail(Assert.java:55)
	at junit.framework.Assert.assertTrue(Assert.java:22)
	at junit.framework.Assert.assertTrue(Assert.java:31)
	at junit.framework.TestCase.assertTrue(TestCase.java:200)
	at org.spdx.tools.CompareSpdxDocsTest.testCompareDocuments(CompareSpdxDocsTest.java:87)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at junit.framework.TestCase.runTest(TestCase.java:177)
	at junit.framework.TestCase.runBare(TestCase.java:142)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:130)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Running org.spdx.tools.SpdxConverterTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.893 sec
Running org.spdx.tools.VerifyTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.56 sec
Running org.spdx.tools.schema.OwlToXSDTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.489 sec
Running org.spdx.tools.GenerateVerificationCodeTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.047 sec

Results :

Failed tests:   testCompareDocuments(org.spdx.tools.CompareSpdxDocsTest)

Tests run: 19, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  36.503 s
[INFO] Finished at: 2021-03-08T14:14:52-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project tools-java: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/rjudge/tempdemo/tools-java/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[goneall]

I was able to duplicate the error - not sure what caused the regression, but it should be an easy fix.

The issue is if there are more than one DocumentDescribes and the order is different, the compare shows this as different when it should be the same (order should not matter).

e.g. SPDXRef-File(./package/foo.c), SPDXRef-Package(glibc) should match SPDXRef-Package(glibc), SPDXRef-File(./package/foo.c)

[goneall]

Update - my initial assessment isn't correct - the order doesn't matter in the compare, that is working correctly.

This will take a while to figure out.

Attached is the spreadsheet with the compares. Everything should match.

What's interesting is that this got introduced when there were no changes to the code itself. I went back and re-ran the CI for a previously passing PR and it now fails.

[goneall]

Here's the spreadsheet with the compare results.
comp.xlsx

[goneall]

I checked out the previous release, and it is also failing with the same error. Very curious. We did do a new release of the license list recently which is loaded during the tests. This is could be the cause of the issue, but it will require some more investigation.

[goneall]

Just tried using an earlier version of the license list - same result. Same code, same data inputs, different results. Extremely curious.

[goneall]

Narrowed it down to a change in the License List data published. The new release includes an additional field licenseTextHtml which is not present in some of the other formats. This is causing the comparison of 2 listed licenses to fail.

This will need to be fixed in the Spdx-Java-Library compare utility and a new version will need to be published before this shows up as fixed.

[rnjudge]

Wow, @goneall! As always, I am impressed and appreciate how fast you got to the bottom of this. Thank you! :)