MathLoadTest_all_0 FAILED in Test_openjdk14_hs_extended.system_aarch64_linux

[d3r3kk]

Test Info
Test Name: MathLoadTest_all_0
Test Duration: 11 sec
Machine: test-aws-ubuntu1804-armv8-1
TRSS link for the test output: https://trss.adoptopenjdk.net/output/test?id=5f0edf557eaab3443ca848c4

Build Info
Build Name: Test_openjdk14_hs_extended.system_aarch64_linux
Jenkins Build start time: Jul 15 2020, 02:39 am
Jenkins Build URL: https://ci.adoptopenjdk.net/job/Test_openjdk14_hs_extended.system_aarch64_linux/140/
TRSS link for the build: https://trss.adoptopenjdk.net/allTestsInfo?buildId=5f0eded87eaab3443ca8477b

Java Version
[0.009s][error][cds] Unable to map CDS archive -- os::vm_allocation_granularity() expected: 65536 actual: 4096
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.2+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.2+12, mixed mode)

This test has been failed 19 times since Jun 25 2020, 01:45 pm
Java Version when the issue first seen
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7-202006252018)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7-202006252018, mixed mode, sharing)
Jenkins Build URL: https://ci.adoptopenjdk.net/job/Test_openjdk14_hs_extended.system_aarch64_linux/122/

The test failed on machine test-aws-ubuntu1804-armv8-1 1 times
The test failed on machine test-aws-rhel76-armv8-3 3 times
The test failed on machine test-aws-rhel76-armv8-4 3 times
The test failed on machine test-packet-ubuntu1604-armv8-1 4 times
The test failed on machine test-aws-rhel76-armv8-2 2 times
The test failed on machine test-packet-ubuntu1604-armv8-2 2 times
The test failed on machine test-aws-rhel76-armv8-6 2 times
The test failed on machine test-aws-rhel76-armv8-1 1 times
The test failed on machine test-aws-rhel76-armv8-5 1 times

[smlambert]

This failure likely still applies to jdk15. @lumpfish - would you be able to take a look at this test running against hotspot to make sure the test material is valid?

Potentially related (though I have not looked at the test source to confirm): eclipse-openj9/openj9#10918 - hotspot answer to math.pow() are 'not correct' (but the Java spec allows for them to be not correct).

[lumpfish]

Yes, I'll take a look.

[lumpfish]

There are similar failures across some of the platforms in both jdk11 and jdk15. The tests are complaining about the precision of various operations being incorrect - -e.g.

jdk15
aarch64
22:19:25  MLT testFailure: testCos(net.adoptopenjdk.test.math.MathAPITest): cos(double)[0] :: expected:<-0.9999876894265599> but was:<-0.99998768942656>
22:19:25  MLT junit.framework.AssertionFailedError: cos(double)[0] :: expected:<-0.9999876894265599> but was:<-0.99998768942656>

jdk11
ppc64_aix
02:22:58  MLT testFailure: testTan(net.adoptopenjdk.test.math.MathAPITest): tan(double)[10] :: expected:<1.2786058599594134> but was:<1.2786058599594137>
02:22:58  MLT junit.framework.AssertionFailedError: tan(double)[10] :: expected:<1.2786058599594134> but was:<1.2786058599594137>

Quoting from the eclipse/openj9 issue eclipse-openj9/openj9#10918:

Actually, it's Hotspot that has the "incorrect" answer, although this is allowed by the spec. Using StrictMath, the answer is 13.966610165238237 on both OpenJ9 and Hotspot.

"Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility is not required."

I haven't found any failures in any of the jdk8 nightly test runs, but I'll do some more pointed testing and see exactly which releases and platforms we do see it on.

So I don't think there's anything wrong with the test case per se, but if the result of an operation cannot be predicted then it's not easy to test it.

[lumpfish]

I've done some analysis and made a proposal for making the test more robust in adoptium/aqa-systemtest#376.

[lumpfish]

This is fixed by adoptium/aqa-systemtest#378