OpenJ9 / OMR fails to build on AIX when using latest XLC v16.1

[groeges]

In OpenJDK jdk branch (jdk13) and hence the jdk13 extensions repository there have been changes to use xlclang/xlclang++ if available on AIX systems.

If XLC 16.1 is installed then xlclang/xlclang++ is provided and used when building OpenJ9/OMR.
When using xlclang/xlclang++ there are quite a lot of compiler/linker options that are not supported anymore and cause the build to fail.

This website shows all the options that are unsupported: https://www.ibm.com/support/knowledgecenter/SSGH3R_16.1.0/com.ibm.xlcpp161.aix.doc/migrate/migrate_to_xlclang.html

The makefiles for OpenJ9 and OMR need to be modified to not use these unsupported options.

[pshipton]

@groeges wrote:
Have just seen that the minimum compiler version for JDK 13 on AIX is now XLC 16.1 (so that the newer xlclang/xlclang++ can be used).

[DanHeidinga]

related OMR issue: eclipse/omr#3775

[vsebe]

related OMR issue: eclipse/omr#3927

[DanHeidinga]

We're likely at the "go / no-go" point for AIX to make JDK 13.

Build is broken so testing hasn't been run yet.
Let's limit the PRs for this going into 0.16 until we're sure we're going to ship it.

[pshipton]

Too late for the 0.16 release now, moving to the next release. Since the next release is next month, we'll try to get AIX working in 0.17 rather than do a late update to 0.16.

[pshipton]

Running a build with all the fixes to date applied and DDR disabled resulted in the following, Illegal instruction. Machine used is a P7.

12:23:07  Compiling 4 files for BUILD_JIGSAW_TOOLS

12:23:44  Optimizing the exploded image

12:23:58  /opt/freeware/bin/bash: line 1: 5505162 Illegal instruction     (core dumped) /home/jenkins/workspace/Build_JDK13_ppc64_aix_Personal/build/aix-ppc64-server-release/jdk/bin/java -Xms64M -Xmx1600M -XX:ThreadStackSize=1536 -XX:+UseSerialGC -Xms32M -Xmx512M -XX:TieredStopAtLevel=1 -cp /home/jenkins/workspace/Build_JDK13_ppc64_aix_Personal/build/aix-ppc64-server-release/buildtools/tools_jigsaw_classes --add-exports java.base/jdk.internal.module=ALL-UNNAMED build.tools.jigsaw.AddPackagesAttribute /home/jenkins/workspace/Build_JDK13_ppc64_aix_Personal/build/aix-ppc64-server-release/jdk > >(/opt/freeware/bin/tee -a /home/jenkins/workspace/Build_JDK13_ppc64_aix_Personal/build/aix-ppc64-server-release/jdk/_optimize_image_exec.log) 2> >(/opt/freeware/bin/tee -a /home/jenkins/workspace/Build_JDK13_ppc64_aix_Personal/build/aix-ppc64-server-release/jdk/_optimize_image_exec.log >&2)
12:23:58  ExplodedImageOptimize.gmk:39: recipe for target '/home/jenkins/workspace/Build_JDK13_ppc64_aix_Personal/build/aix-ppc64-server-release/jdk/_optimize_image_exec.marker' failed

[pshipton]

@gita-omr the Illegal instruction happens when running java -version, but not java -Xint -version. I have the JVM which I can make accessible, can somebody pls take a look.
@zl-wang

[gita-omr]

Sure, we'll take a look as soon as somebody is available.

[pshipton]

I don't expect this is going to make the 0.17 release, so moving it out.

[gita-omr]

@keithc-ca I heard you looked at the 'java -version' crash. Please let us know if we can help.

[keithc-ca]

@keithc-ca I heard you looked at the 'java -version' crash. Please let us know if we can help.

I have not looked at that.

[gita-omr]

@keithc-ca I heard you looked at the 'java -version' crash. Please let us know if we can help.

I have not looked at that.

Sure, we will look into it

[pshipton]

@zl-wang is somebody working on a better fix like #7034? This change was reverted. If we get a fix for this, temporarily disable the DDR support, and remove the -bnorwexec link option, we can get a build and test run.

[pshipton]

Issue opened at Adopt for the Hotspot crash adoptium/temurin-build#1327

[zl-wang]

@pshipton we asked xlC team, and they cannot fix it (linking in libc++.a by default), since the same command can be used for xlC build (as opposed to xlclang++ build) which doesn't need libc++.a.

we have to decide if libc++.a should be statically linked in as well (it is not available on most AIX installations).

[pshipton]

@zl-wang it seems we have the start of a solution #7034 (comment)
Yes, it should be statically linked.

[pshipton]

I ran an AIX build/test using xlc 13 with the OpenJDK 13.0.0 level. Besides the OpenJ9 change to compile with xlc 13, this requires the harfbuzz fix which was applied to openjdk12 to be applied to openjdk13. The standard functional and system testing passed.
https://ci.eclipse.org/openj9/view/Pipelines/job/Pipeline-Release-Build/77

I'll have to retry this with the 13.0.1 level when it promotes. Using xlc 13 is fragile, as OpenJDK changes could break this at any time. i.e. even if it works for 13.0.1, it may not work for 13.0.2 in January. We can also try this with the jdk14 code to date.

I'll also try a xlc 16 build. This build won't have DDR support so test failures are expected, but we can see if anything else fails.

[pshipton]

The xlc 13 build is https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK13_ppc64_aix_Personal/31/OpenJ9-JDK13-ppc64_aix-20191016-215625.tar.gz

and the xlc 16 build is https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK13_ppc64_aix_Personal/33/OpenJ9-JDK13-ppc64_aix-20191017-153805.tar.gz

these are created from equivalent source, and can be used to compare performance.

The top level build for xlc 16 is https://ci.eclipse.org/openj9/job/Pipeline-Release-Build/79
Test results:
sanity.functional: only a ddr failure
extended.functional: only ddr failures
sanity.system: passed
extended.system: passed

[pshipton]

With xlc 16 we only got the expected DDR failures, all the other standard testing passed.

[pshipton]

xlc 13 with jdk13.0.1 compiled fine. I didn't bother with testing.
https://ci.eclipse.org/openj9/job/Pipeline-Release-Build/80

[zl-wang]

I have a preliminary performance tests of xlc13/16 builds (performance of C/C++ code, that is: interpreter and JIT-compiler specifically). I used javac helloWorld as the test case: -Xint and -Xjit:count=0,optLevel=hot -Xnoaot (to exercise JIT-compiler).

-Xint run, xlc16 build looks 15% better;
JIT-compiler run, xlc13 build looks 8% better.

[zl-wang]

Overall, I felt xlclang++ build is fine perf-wise. Although, it has the glitch that I have to copy libc++.a and related to where I ran (and set LIBPATH).

[pshipton]

As per #7499, we are statically linking libc++

[zl-wang]

you meant the xlc16 build link you gave above is supposed to be built with static linking already?

[pshipton]

you meant the xlc16 build link you gave above is supposed to be built with static linking already?

No, my earlier build wasn't statically linked. We are delivering a change to statically link in future builds. I'll provide a link to the build when it's ready.

[pshipton]

Here is a statically linked xlc 16 build https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK13_ppc64_aix_Personal/36/OpenJ9-JDK13-ppc64_aix-20191018-143648.tar.gz

[pshipton]

It's more than just the JIT which is linking to libc++ now, see #7499 (comment), plus I don't think the JIT is statically linking properly yet, see #7499 (comment)

[pshipton]

Seems jdk14 no longer supports compilation with xlc 13
configure: error: xlclang++ version output check failed, output: /home/u0020236/workspace/Build_JDKnext_ppc64_aix_Personal/build/.configure-support/generated-configure.sh: line 40731: xlclang++: command not found

[pshipton]

#8055 is merged, DDR is enabled on AIX Java 13+, this is now complete, although #7579 is still open,