jdk11 jdk_compile failures on macos12

[sophia-guo]

56 compiler/aot/* tests failed on test-macincloud-macos1201-x64-1 only with message:

15:02:26  Preparing compiled binary (0 ms)
15:02:26  Creating binary: libHelloWorld.so.o (9 ms)
15:02:26  Creating shared library: libHelloWorld.so (19 ms)
15:02:26  Exception in thread "main" java.lang.InternalError: ld: dynamic main executables must link with libSystem.dylib for architecture x86_64
15:02:26  	at jdk.aot@11.0.15-beta/jdk.tools.jaotc.Linker.link(Linker.java:142)
15:02:26  	at jdk.aot@11.0.15-beta/jdk.tools.jaotc.Main.run(Main.java:262)
15:02:26  	at jdk.aot@11.0.15-beta/jdk.tools.jaotc.Main.run(Main.java:133)
15:02:26  	at jdk.aot@11.0.15-beta/jdk.tools.jaotc.Main.main(Main.java:89)
15:02:26  
15:02:26  Command line: [/Users/jenkins/workspace/grinder_sandbox_new/openjdkbinary/j2sdk-image/Contents/Home/bin/../bin/java -cp /Users/jenkins/workspace/grinder_sandbox_new/aqa-tests/TKG/output_1650394746878/hotspot_custom_1/work/classes/0/compiler/aot/verification/ClassAndLibraryNotMatchTest.d:/Users/jenkins/workspace/grinder_sandbox_new/aqa-tests/openjdk/openjdk-jdk/test/hotspot/jtreg/compiler/aot/verification:/Users/jenkins/workspace/grinder_sandbox_new/aqa-tests/TKG/output_1650394746878/hotspot_custom_1/work/classes/0/test/lib:/Users/jenkins/workspace/grinder_sandbox_new/aqa-tests/TKG/output_1650394746878/hotspot_custom_1/work/classes/0:/Users/jenkins/workspace/grinder_sandbox_new/jvmtest/openjdk/jtreg/lib/javatest.jar:/Users/jenkins/workspace/grinder_sandbox_new/jvmtest/openjdk/jtreg/lib/jtreg.jar:/Users/jenkins/workspace/grinder_sandbox_new/jvmtest/openjdk/jtreg/lib/junit.jar:/Users/jenkins/workspace/grinder_sandbox_new/jvmtest/openjdk/jtreg/lib/testng.jar:/Users/jenkins/workspace/grinder_sandbox_new/jvmtest/openjdk/jtreg/lib/jcommander.jar -ea -esa -Xmx512m -XX:-UseCompressedOops -cp . -XX:+UnlockExperimentalVMOptions -XX:+UseAOT -XX:AOTLibrary=./libHelloWorld.so HelloWorld ]
15:02:26  STDERR:
15:02:26   stdout: [error opening file: dlopen(./libHelloWorld.so, 0x0001): tried: '/Users/jenkins/workspace/grinder_sandbox_new/openjdkbinary/j2sdk-image/Contents/Home/lib/server/././libHelloWorld.so' (no such file), '/Users/jenkins/workspace/grinder_sandbox_new/openjdkbinary/j2sdk-image/Contents/Home/lib/server/.././libHelloWorld.so' (no such file), '/Users/jenkins/workspace/grinder_sandbox_new/openjdkbinary/j2sdk-image/Contents/Home/bin/../lib/jli/./libHelloWorld.so' (no such file), './libHelloWorld.so' (no such file), '/usr/local/lib/libHelloWorld.so' (no such file), '/usr/lib/libHelloWorld.so' (no such file), '/Users/jenkins/workspace/grinder_sandbox_new/aqa-tests/TKG/output_1650394746878/hotspot_custom_1/work/scratch/1/libHelloWorld.so' (no such file), '/usr/local/lib/libHelloWorld.so' (no such file), '/usr/lib/libHelloWorld.so' (no such file)
15:02:26  ];
15:02:26   stderr: []
15:02:26   exitValue = 1
15:02:26  
15:02:26  java.lang.RuntimeException: Expected to get exit value of [0]

https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_extended.openjdk_x86-64_mac_testList_1/41/
https://trss.adoptium.net/deepHistory?testId=625744b172761a4c5c12015c

[sophia-guo]

Same on test-macincloud-macos1201-x64-2

[sophia-guo]

Two jdk_net tests failed on macos12 with different error message. https://trss.adoptium.net/deepHistory?testId=62b789e3250c3c428c3ad570

[sophia-guo]

We would like to take those two machines offline before the issues is fixed (either update the test cases or update machine configuration?) to avoid re-triage again and again. Looks like https://ci.adoptopenjdk.net/computer/test-macincloud-macos1201-x64-1/ is offline due to some other updates.

The interesting thing is both machines are labeled as macos10.10 https://ci.adoptopenjdk.net/label/macos10.10/, which is incorrect. Might be updated from macos10.10 to macos10.12?

[sophia-guo]

jdk_net with jdk18 failed https://ci.adoptopenjdk.net/job/Test_openjdk18_hs_extended.openjdk_x86-64_mac_testList_1/41/

history shows only fail on macos12
https://trss.adoptium.net/deepHistory?testId=62c11de6250c3c428c63099a

[sophia-guo]

Latest failing builld
https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_extended.openjdk_x86-64_mac_testList_1/55/#showFailuresLink

[sxa]

jdk_net with jdk18 failed https://ci.adoptopenjdk.net/job/Test_openjdk18_hs_extended.openjdk_x86-64_mac_testList_1/41/

history shows only fail on macos12 https://trss.adoptium.net/deepHistory?testId=62c11de6250c3c428c63099a

That looks like a separate failure from the one mentioned in the description, so perhaps it should have its own issue

[sxa]

The interesting thing is both machines are labeled as macos10.10 https://ci.adoptopenjdk.net/label/macos10.10/, which is incorrect. Might be updated from macos10.10 to macos10.12?

Agreed - that's an error. Updated to macos12

[sophia-guo]

There is no JBS issue reported yet. However feels like it is the test issue. As New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286). Similar issue can't link object file with ld on macOS . To make those tests work with macos11+ command needs to be updated Command line: [/usr/bin/ld -v] with flag -lSystem. and -macosx_version_min. Note, those tests are not in 17+ any more.

[sxa]

Does the failure show on the aarch64 builds which are on macos11.0.1? The above would suggest that it would affect those too.

[sxa]

Looking at https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_extended.openjdk_aarch64_mac_testList_1/17/testReport/ it looks like those tests are not run on aarch64 so the consistent option would seem to be be to exclude them on the x86-64 platform too.

[sophia-guo]

Does the failure show on the aarch64 builds which are on macos11.0.1? The above would suggest that it would affect those too.

Do we have macos11? I didn't find one https://ci.adoptopenjdk.net/label/ci.role.test&&hw.arch.x86&&%28sw.os.osx%7C%7Csw.os.mac%29/

Ignore this, i didn't realize @sxa mentioned aarch64 macos11

[sophia-guo]

@sxa those aot tests only run if VM supports AOT. Does jdk11 arch64 mac support AOT? If no tests will skip.

[sxa]

@sxa those aot tests only run if VM supports AOT. Does jdk11 arch64 mac support AOT? If no tests will skip.

Interesting - apparently not:

19:03:40  checking if aot should be enabled... no

It says yes for x64.

[sxa]

jdk_net with jdk18 failed https://ci.adoptopenjdk.net/job/Test_openjdk18_hs_extended.openjdk_x86-64_mac_testList_1/41/
history shows only fail on macos12 https://trss.adoptium.net/deepHistory?testId=62c11de6250c3c428c63099a

That looks like a separate failure from the one mentioned in the description, so perhaps it should have its own issue

Yeah these seem to be IP address/socket issues. One is giving Execution failed: main threw exception: java.lang.RuntimeException: invalidhost.invalid should throw UnknownHostException! so this will definitely need separate diagnosis outside this issue.

[sophia-guo]

This keeps failing tests during the latest two releases.

[sophia-guo]

#4326 (comment)

This feels like a upstream issue. Would need help to confirm from and fix in upstream.

[sophia-guo]

Same issue with github runner macos-latest(12). https://github.com/sophia-guo/runaqaTest/actions/runs/4136572808/jobs/7150623636