AQAvit Triage for January 2023: jdk8u362-b09

[smlambert]

• https://ci.adoptopenjdk.net/job/build-scripts/job/release-openjdk8-pipeline/7/
• https://ci.adoptopenjdk.net/job/build-scripts/job/release-openjdk8-pipeline/8/

Unfortunately, due to deleting dry runs on Jenkins and subsequent job IDs starting at 1 again, and adoptium/aqa-test-tools#767, none of the 3 release-jdk8 pipelines are in the TRSS DB (since those 'unique' IDs are already inserted into the DB while monitoring the dry runs). So they will not show up in any of the TRSS views.

This triage will have to be done 'blind' (with no good access to deep history), but by navigating Jenkins jobs.

Platforms	Details	Post triage Status
x86-64_linux	details	✅
x86-64_mac	details	✅
x86-64_windows	details	✅
aarch64_linux	details	✅
x86-32_windows	details	✅
ppc64_aix	details	✅
ppc64le_linux	details	✅
x86-64_alpine-linux	details	✅
x86-64_solaris	details	✅
sparcv9_solaris	details	✅ same set of failures as October, so will ship it - they need to be assessed/excluded (see #4150)
arm_linux	details	✅

[smlambert]

x86-64_windows

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - pass
sanity.perf - pass
sanity.system - pass
extended.system - pass
extended.openjdk - fail, jdk_security1 is known, non-blocking failure, others rerun in Grinder/6388 - jdk_jfr passes, jdk_management still fails (with port in use), rerun just JMXStartStopTest in Grinder/6397 - passes

jdk_security1 failures:
java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
java/security/cert/pkix/policyChanges/TestPolicy.java

jdk_management failure:
sun/management/jmxremote/startstop/JMXStartStopTest.java

[JMXStartStopDoSomething] Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 57073; nested exception is: 
[JMXStartStopDoSomething] 	java.net.BindException: Address already in use: JVM_Bind
[JMXStartStopDoSomething] sun.management.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 57073; nested exception is: 
[JMXStartStopDoSomething] 	java.net.BindException: Address already in use: JVM_Bind
[JMXStartStopDoSomething] 	at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:480)
[JMXStartStopDoSomething] 	at sun.management.Agent.startRemoteManagementAgent(Agent.java:207)

jdk_jfr failure: 
jdk/jfr/event/runtime/TestNetworkUtilizationEvent.java

• jdk_security1 failures are known and non-blocking (see jdk_security1 FAILED in Test_openjdk8_hs_extended.openjdk (all platforms) #4244)

extended.perf seems to be running and passes but does not need to be (as part of a release / AQAvit verification)

[smlambert]

x86-64_linux

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - pass
sanity.perf - pass
sanity.system - pass
extended.system - pass
extended.openjdk - fail, rerun at Grinder/6375 - jdk_security1 tests are known issue and non-blocking, other tests pass

jdk_security1: known and non-blocking (aqa-tests#4244)
java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
java/security/cert/pkix/policyChanges/TestPolicy.java

----

jdk_nio:
java/nio/file/Files/probeContentType/Basic.java
Expected "text/html" but obtained "text/plain"
java.lang.RuntimeException: Test failed!
	at Basic.main(Basic.java:136)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.lang.Thread.run(Thread.java:750)

JavaTest Message: Test threw exception: java.lang.RuntimeException: Test failed!

jdk_management:
java/lang/management/OperatingSystemMXBean/TestSystemLoadAvg.sh
12:19:42  /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/../bin/java -classpath /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/aqa-tests/TKG/output_16740599688272/jdk_management_0/work/classes/0/java/lang/management/OperatingSystemMXBean GetSystemLoadAverage
12:19:42  TEST FAILED: TestSystemLoadAvg test failed 5 runs
12:19:42  STDERR:
12:19:42  Exception in thread "main" java.io.IOException: Cannot run program "/usr/bin/uptime": error=2, No such file or directory
12:19:42  	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
12:19:42  	at GetSystemLoadAverage.checkLoadAvg(GetSystemLoadAverage.java:90)
12:19:42  	at GetSystemLoadAverage.main(GetSystemLoadAverage.java:64)
12:19:42  Caused by: java.io.IOException: error=2, No such file or directory
12:19:42  	at java.lang.UNIXProcess.forkAndExec(Native Method)
12:19:42  	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
12:19:42  	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
12:19:42  	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

----

jdk_rmi:
sun/rmi/rmic/defaultStubVersion/run.sh

12:30:55  STDERR:
12:30:55  
12:30:55  ACTION: shell -- Failed. Execution failed: exit code 127
12:30:55  REASON: User specified action: run shell run.sh 
12:30:55  TIME:   1.066 seconds
12:30:55  messages:
12:30:55  command: shell run.sh
12:30:55  reason: User specified action: run shell run.sh 
12:30:55  elapsed time (seconds): 1.066

----

jdk_tools:
tools/launcher/VersionCheck.java

Actual:
/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: 1: /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: �ELF\u0002\u0001\u0001\u0003: not found
/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: 2: /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: Syntax error: "(" unexpected
Error: /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo fails -J-fullversion comparison
Expected:
openjdk full version "1.8.0_362-b09"
Actual:
/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: 1: /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: �ELF\u0002\u0001\u0001\u0003: not found
/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: 2: /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/idlj.debuginfo: Syntax error: "(" unexpected
Testing (-J-version): /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/openjdkbinary/j2sdk-image/jdk8u362-b09/bin/jrunscript
Version Test: 86

[smlambert]

x86-64_mac

sanity.functional - pass
extended.functional - pass
special.functional -
sanity.openjdk - pass
sanity.perf - pass
sanity.system - pass
extended.system - pass
extended.openjdk - fail with known, non-blocking failures (see #4244)

TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java

[smlambert]

BLOCKED by adoptium/infrastructure#2934 SL/Jan24th creative workaround by running on temurin-compliance Jenkins server (suspect machine issue for ci-jenkins-pipelines/issues/553)

alpine-linux test jobs can not run due to long-standing issue adoptium/infrastructure#2934
DETECTED_JDK_VERSION value is 17, settled JDK_VERSION value is 8. JDK_VERSION value does not match.

Will need to kick off an AQA_Test_Pipeline with SDK_RESOURCE = customized to hope for test results.
Nope, that will not allow us to get around the issue (same failures using customized):
https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/126

SL/Jan24th: Launching via
https://ci.eclipse.org/temurin-compliance/job/AQA_Test_Pipeline/988/ instead, and tests are queued up for their turn on the limited set of machines.

sanity.functional - pass
extended.functional - pass
special.functional - pass

sanity.system - pass
extended.system - pass

sanity.perf - pass

sanity.openjdk - pass
extended.openjdk - failed but with the known non-blockers

Failed test cases: 
TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java
Test results: passed: 172; failed: 2

[smlambert]

aarch64_linux

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - fail, tests timing out on test-docker-debian11-armv8-1, rerun at https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_aarch64_linux/786 - pass

sanity.perf - pass
sanity.system - pass
extended.system - fail, due to being terminated mid-run (see adoptium/infrastructure#2888), rerun at: https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.system_aarch64_linux/774 - fails with LoadDB_5m test, rerun in Grinder/6441 and also in Grinder/6442 - passes

extended.openjdk - fail, but with known non-blocking issue (see #4244)

[smlambert]

x86-32_windows

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - pass
sanity.perf - pass
sanity.system - pass
extended.system - pass
extended.openjdk - fail, but with known non-blocking issue #4244

TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java

[smlambert]

ppc64_aix

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - pass
sanity.perf - pass
sanity.system - pass
extended.system - pass
extended.openjdk - fail

hotspot_jre:
TEST: runtime/SharedArchiveFile/DefaultUseWithClient.java
Unrecognized option: -client
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

jdk_security1 (known issue):
TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java

Possibly related: https://bugs.openjdk.org/browse/JDK-8297141 - in any event, this testcase should not be running against ppc64_aix (see https://github.com/adoptium/jdk8u/blob/master/hotspot/test/runtime/SharedArchiveFile/DefaultUseWithClient.java#L39-L45), not sure why it is, but not valid there. Consider the failure non-blocking. Raised #4270.

[smlambert]

ppc64le_linux

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - pass
sanity.perf - pass
sanity.system - pass
extended.system - pass
extended.openjdk - fail with known, non-blocking test issue #4244

jdk_security1:
TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java

[smlambert]

x86-64_solaris

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - pass
sanity.perf - pass
sanity.system - pass
extended.system - pass
extended.openjdk - fail, jdk_security1 known non-blocking issue, jdk_security3 failing on test-equinix_esxi-solaris10-x64-1-NEW, rerun on different machine in Grinder/6437 and again Grinder/6447 passes

jdk_security1 (known issue):
TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java

jdk_security3:
TEST: sun/security/ssl/SSLSocketImpl/AsyncSSLSocketClose.java

[smlambert]

sparcv9_solaris

sanity.functional - pass
extended.functional - pass
special.functional - pass
sanity.openjdk - fail (seen as a regression between July and October 2022 releases, but still released this platform. We need to chase down the root of this problem, related: #4150 (comment).
java/math/BigInteger/LargeValueExceptions.java
sanity.perf - pass
sanity.system - pass
extended.system - fail, OAuthTest, retry on the other machine, test-siteox-solaris10u11-sparcv9-1 in Grinder/6448
extended.openjdk - aborted, run in parallel both child jobs get aborted after 10hrs, rerun both with 50hr timeout to get through it. https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_sparcv9_solaris_testList_0/69
and https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_sparcv9_solaris_testList_1/66

jdk_security1 (known and non-blocking):
TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java

jdk_security3:
TEST: javax/net/ssl/Stapling/HttpsUrlConnClient.java
TEST: javax/net/ssl/Stapling/SSLSocketWithStapling.java
TEST: javax/net/ssl/TLSCommon/ConcurrentClientAccessTest.java
TEST: javax/net/ssl/TLSCommon/TestSessionLocalPrincipal.java
TEST: sun/security/pkcs11/fips/ClientJSSEServerJSSE.java
TEST: sun/security/pkcs11/KeyStore/ClientAuth.sh
Test results: passed: 610; failed: 3; error: 3

hotspot_jre:
TEST: compiler/8209951/TestCipherBlockChainingEncrypt.java

jdk_jfr (pass on :
TEST: jdk/jfr/event/gc/stacktrace/TestG1OldAllocationPendingStackTrace.java
TEST: jdk/jfr/javaagent/TestLoadedAgent.java
TEST: jdk/jfr/jcmd/TestJcmdSaveToFile.java
TEST: jdk/jfr/jcmd/TestJcmdStartWithOptions.java
TEST: jdk/jfr/jcmd/TestJcmdStartWithSettings.java
TEST: jdk/jfr/jcmd/TestJcmdStopInvalidFile.java
TEST: jdk/jfr/jcmd/TestJcmdStopReadOnlyFile.java
Test results: passed: 411; failed: 6; error: 1

jdk/jfr/jcmd/* testcases occasionally coredump executing jcmd (not in all cases, appears to be in cases where JFR.stop is used, needs further investigation):

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xffffffff7d769804, pid=20473, tid=0x0000000000000014
#
# JRE version: OpenJDK Runtime Environment (8.0_362-b09) (build 1.8.0_362-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.362-b09 mixed mode solaris-sparc compressed oops)
# Problematic frame:
# V  [libjvm.so+0x769804][thread 27 also had an error]
[thread 29 also had an error][thread 25 also had an error]

[thread 28 also had an error]
[thread 26 also had an error]
  void JfrNetworkInterfaceName::serialize(JfrCheckpointWriter&)+0xec
#
# Core dump written. Default location: /export/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_sparcv9_solaris_testList_1/aqa-tests/TKG/output_16744972695799/jdk_jfr_0/work/scratch/core or core.20473

Try some reruns at TC_Grinder/3116

Also noting the performance degradation between run from September 10th to present day, what changed (new machine config? tests starting to fail, so hitting timeouts? needs investigation):

SL/Jan25: noting that we released this platform in October with the same set of failures that have not been addressed or excluded, and while we need to do this (fix or exclude), rather than carry forward each release, we will proceed with releasing this platform despite these failures (related: #4150).
SL/Feb22: created #4359

[smlambert]

arm_linux (see #4258 (comment))

[zdtsw]

arm_linux

sanity.functional https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.functional_arm_linux/285/

 TOTAL: 3   EXECUTED: 1   PASSED: 1   FAILED: 0   DISABLED: 1   SKIPPED: 1
00:02:01.494  ALL TESTS PASSED

extended.functional https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.functional_arm_linux/284/

 TOTAL: 4   EXECUTED: 2   PASSED: 2   FAILED: 0   DISABLED: 0   SKIPPED: 2
00:02:00.024  ALL TESTS PASSED

special.functional https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_special.functional_arm_linux/81/

TOTAL: 170   EXECUTED: 53   PASSED: 53   FAILED: 0   DISABLED: 0   SKIPPED: 117
00:06:14.311  ALL TESTS PASSED

sanity.openjdk https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_arm_linux/474/

FAILED test targets:
08:10:48.229  	jdk_util_2  
08:10:48.229  
08:10:48.229  TOTAL: 12   EXECUTED: 4   PASSED: 3   FAILED: 1   DISABLED: 0   SKIPPED: 8

^re-run grinder: https://ci.adoptopenjdk.net/job/Grinder/6408/console --- PASSED

sanity.perf https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.perf_arm_linux/297/

TOTAL: 9   EXECUTED: 5   PASSED: 5   FAILED: 0   DISABLED: 3   SKIPPED: 1
01:08:55.829  ALL TESTS PASSED

sanity.system https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.system_arm_linux/271/

FAILED test targets:
01:10:09.828  	TestJlmRemoteClassAuth_2 
01:10:09.828  	TestJlmRemoteThreadNoAuth_2 
01:10:09.828  
01:10:09.828  TOTAL: 60   EXECUTED: 21   PASSED: 19   FAILED: 2   DISABLED: 12   SKIPPED: 27

^re-run grinder: https://ci.adoptopenjdk.net/job/Grinder/6409/console
^more re-run ginder: https://ci.adoptopenjdk.net/job/Grinder/6411/console https://ci.adoptopenjdk.net/job/Grinder/6413/console
PASSED

extended.system https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.system_arm_linux/271/

TOTAL: 38   EXECUTED: 19   PASSED: 19   FAILED: 0   DISABLED: 3   SKIPPED: 16
00:50:31.982  ALL TESTS PASSED

extended.openjdk https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_arm_linux/75/
-https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_arm_linux_testList_0/67/

TOTAL: 9   EXECUTED: 9   PASSED: 9   FAILED: 0   DISABLED: 0   SKIPPED: 0
02:38:36.135  ALL TESTS PASSED

-https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_arm_linux_testList_1/67/

FAILED test targets:
02:57:50.567  	hotspot_jre_2 
02:57:50.567  	jdk_jfr_2 
02:57:50.567  
02:57:50.567  TOTAL: 72   EXECUTED: 8   PASSED: 6   FAILED: 2   DISABLED: 12   SKIPPED: 52

^re-run grinder: https://ci.adoptopenjdk.net/job/Grinder/6415/console
node test-sxa-armv7l-ubuntu2004-odroid-1 just died in the mid of grinder, had to abort it and change to a new node.
https://ci.adoptopenjdk.net/job/Grinder/6420/console

• SL/Jan23: hotspot_jre testcase compiler/c2/Test8202414.java is same as https://bugs.openjdk.org/browse/JDK-8251152 and should be excluded with reference to JDK-8251152
• SL/Jan23: jdk_jfr testcases appear to pass on test-sxa-armv7l-ubuntu2004-odroid-2 on Dec 10th (according to Deep History), rerun in Grinder/6451 - had to abort but that is the only machine where jdk_jfr tests have ever passed

22:49:26  XML output with verification to /ssd/jenkins/workspace/Grinder/aqa-tests/TKG/output_1674532162630/jdk_jfr_2/work
23:01:03  Cannot contact test-sxa-armv7l-ubuntu2004-odroid-1: java.lang.InterruptedException

[wen 24th Jan morning]:
set to use test-sxa-armv7l-ubuntu2004-odroid-1 to re-run https://ci.adoptopenjdk.net/job/Grinder/6455/console
https://ci.adoptopenjdk.net/job/Grinder/6456/console both hanging with

JavaTest Message: Problem cleaning up the following threads:
Thread-0
at Test6891750.test(Test6891750.java:69)
at Test6891750.run(Test6891750.java:88) 

-https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_arm_linux_testList_2/66/

FAILED test targets:
01:25:35.087  	jdk_other_2 
01:25:35.087  	jdk_security1_2 
01:25:35.087  	jdk_security3_2 
01:25:35.087  	jdk_management_2 
01:25:35.087  
01:25:35.087  TOTAL: 9   EXECUTED: 9   PASSED: 5   FAILED: 4   DISABLED: 0   SKIPPED: 0

^re-run grinder: https://ci.adoptopenjdk.net/job/Grinder/6407/console
^more re-run grinder: https://ci.adoptopenjdk.net/job/Grinder/6414/console
node test-sxa-armv7l-ubuntu2004-odroid-1 just died in the mid of grinder, had to abort it and change to a new node.
https://ci.adoptopenjdk.net/job/Grinder/6419/console
[wen 24th Jan morning] still failed:
TEST: javax/xml/jaxp/common/8144593/ValidationWarningsTest.java
TEST: java/security/cert/CertPathBuilder/targetConstraints/BuildEEBasicConstraints.java
TEST: java/security/cert/pkix/policyChanges/TestPolicy.java
TEST: javax/net/ssl/ALPN/SSLSocketAlpnTest.java
TEST: sun/security/ssl/X509TrustManagerImpl/PKIXExtendedTM.java
TEST: sun/security/ssl/X509TrustManagerImpl/SelfIssuedCert.java

• SL/Jan24: been told that losing connection to the test-sxa-armv7l-ubuntu2004-odroid-2 and -1 machines often resolves itself if you leave it alone, I will launch another Grinder to try and run through the jdk_jfr_2 target at Grinder/6468 - passes
  It is the last target to be able to pass to mark this platform as "done".

[smlambert]

Finished creating related issues, able to close this issue now.