8307858: [REDO] JDK-8307194 Add make target for optionally building a complete set of all JDK and hotspot libjvm static libraries

[jianglizhou]

Original description for JDK-8307194 change:

This PR is branched from the makefile changes for https://bugs.openjdk.org/browse/JDK-8303796 and contains the following for handling the JDK/hotspot static libraries:

• Build hotspot libjvm.a and JDK static libraries for static-libs-image/static-libs-bundles targets; This change does not affect the graal-builder-image target

• For libjvm.a specifically, exclude operator_new.o

• Filter out "external" .o files (those are the .o files included from a different JDK library and needed when creating the .so shared library only) from JDK .a libraries; That's to avoid linker failures caused by duplicate symbols

  • For libjli.a: Not include inflate.o inftrees.o inffast.o zadler32.o zcrc32.o zutil.o (compiled from zlib sources) if zlib is built as JDK bundled
  • For libawt_xawt.a and libawt_head.a: Not include systemScale.o, since it's provided in libawt.a

• Handle long arguments case for static build in make/common/NativeCompilation.gmk

• Address @erikj79's comment in 8303796: Optionally build fully statically linked JDK image #13709 (comment) for LIBJLI_STATIC_EXCLUDE_OBJS

---

Updates to address build failures reported on macosx- platforms:

• For gcc/clang, when building a static library first partially link (using the -r linking option) all object files into one object. The output object file from the partial linking is then passed to ar to create the static library.

The original change for JDK-8307194 used @argument_file for all platforms when dealing with long arguments to ar, which caused failures on macosx- builds. On darwin (https://www.unix.com/man-page/osx/1/ar/), ar does not support @argument_file. The updated change avoids using @argument_file for ar.

The partial linking change is done in make/common/NativeCompilation.gmk. The flag related change is done in make/autoconf/flags-ldflags.m4 mainly.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8307858: [REDO] JDK-8307194 Add make target for optionally building a complete set of all JDK and hotspot libjvm static libraries (Enhancement - P4)

Reviewers

• Erik Joelsson (@erikj79 - Reviewer)

Contributors

• Erik Joelsson <erikj@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/14064/head:pull/14064
$ git checkout pull/14064

Update a local copy of the PR:
$ git checkout pull/14064
$ git pull https://git.openjdk.org/jdk.git pull/14064/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 14064

View PR using the GUI difftool:
$ git pr show -t 14064

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/14064.diff

Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back jiangli! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@jianglizhou The following labels will be automatically applied to this pull request:

• build
• client
• core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 08: Full (af271e2c)
• 07: Full (28585b84)
• 06: Full (c664c601)
• 05: Full (1c2f75dc)
• 04: Full - Incremental (93366e38)
• 03: Full - Incremental (695bda5e)
• 02: Full - Incremental (6a321bd1)
• 01: Full (fb945210)
• 00: Full (8af1ec2f)

[erikj79]

I ran this patch in our internal build and test system and got failures on macos and windows. I think I know the cause for the mac failure, but the Windows failure I don't know. It looks like the same error I saw before after the original patch went in.

[2023-05-19T20:51:31,466Z] c:\sb\prod\1684529071\workspace\open\src\hotspot\share\compiler\disassembler.cpp(792): error C2220: the following warning is treated as an error
[2023-05-19T20:51:31,466Z] c:\sb\prod\1684529071\workspace\open\src\hotspot\share\compiler\disassembler.cpp(792): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
[2023-05-19T20:51:31,466Z] lib/CompileJvm.gmk:152: recipe for target '/cygdrive/c/sb/prod/1684529071/workspace/build/windows-x64-open/hotspot/variant-server/libjvm/objs/static/disassembler.obj' failed
[2023-05-19T20:51:31,466Z] make[3]: *** [/cygdrive/c/sb/prod/1684529071/workspace/build/windows-x64-open/hotspot/variant-server/libjvm/objs/static/disassembler.obj] Error 1

[erikj79]

This command line is missing $$($1_SYSROOT_LDFLAGS) which is causing it to fail in our builds with:

ld: library not found for -lSystem

[erikj79]

This is the mac failure, sorry if that wasn't clear.

[jianglizhou]

Thanks @erikj79. Could you please help provide some more info on the build failure:

Which macOs version ran into the build issue? My mac is on Ventura 13.3.1 (a). It builds successfully for the static-libs-image target.

Does it fail when partially linking libjvm_relocatable.o only, or it fails when partially linking other native libraries as well on macosx? Could you please share the partial linking command, e.g. hotspot/variant-server/libjvm/gtest/objs/static/BUILD_GTEST_LIBJVM_partial_link.cmdline?

[jianglizhou]

Thanks @erikj79. Could you please help provide some more info on the build failure:

Which macOs version ran into the build issue? My mac is on Ventura 13.3.1 (a). It builds successfully for the static-libs-image target.

Does it fail when partially linking libjvm_relocatable.o only, or it fails when partially linking other native libraries as well on macosx? Could you please share the partial linking command, e.g. hotspot/variant-server/libjvm/gtest/objs/static/BUILD_GTEST_LIBJVM_partial_link.cmdline?

Additional info, following is the partial linking command from my macosx-x86_64 build:

/usr/bin/g++ -m64 -r -o /.../github/JDK-8307858/build/linux-x86_64-server-release/hotspot/variant-server/libjvm/gtest/objs/static/libjvm_relocatable.o @/.../github/JDK-8307858/build/linux-x86_64-server-release/hotspot/variant-server/libjvm/gtest/objs/static/_BUILD_GTEST_LIBJVM_objectfilenames.txt

[erikj79]

It fails because we are using a "devkit" and not an installed Xcode. The SYSROOT flags must always be added to any compiler or link command lines.

[jianglizhou]

I ran this patch in our internal build and test system and got failures on macos and windows.

Thanks a lot, @erikj79! I was going to ask your help for testing the patch.

I think I know the cause for the mac failure, but the Windows failure I don't know. It looks like the same error I saw before after the original patch went in.

[2023-05-19T20:51:31,466Z] c:\sb\prod\1684529071\workspace\open\src\hotspot\share\compiler\disassembler.cpp(792): error C2220: the following warning is treated as an error
[2023-05-19T20:51:31,466Z] c:\sb\prod\1684529071\workspace\open\src\hotspot\share\compiler\disassembler.cpp(792): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
[2023-05-19T20:51:31,466Z] lib/CompileJvm.gmk:152: recipe for target '/cygdrive/c/sb/prod/1684529071/workspace/build/windows-x64-open/hotspot/variant-server/libjvm/objs/static/disassembler.obj' failed
[2023-05-19T20:51:31,466Z] make[3]: *** [/cygdrive/c/sb/prod/1684529071/workspace/build/windows-x64-open/hotspot/variant-server/libjvm/objs/static/disassembler.obj] Error 1

The warning error looks the same as https://bugs.openjdk.org/browse/JDK-8307858?focusedCommentId=14581027&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14581027. Sorry I didn't look closely earlier. I just took a look of

jdk/src/hotspot/share/compiler/disassembler.cpp

Line 792 in 939344b

lib_offset = jvm_offset = strlen(buf);

:

#ifdef STATIC_BUILD
char* p = strrchr(buf, '/');
*p = '\0';
strcat(p, "/lib/");
lib_offset = jvm_offset = strlen(buf);
#else

The code is only enabled for the static build, which is why the issue doesn't show up in regular JDK build. strlen return value is size_t. I'll work on a fix.

Could you please help fix the mac failure as you have already looked into it. Thanks!

[jianglizhou]

/contributor add @erikj79

Thanks for running tests!!

[openjdk]

@jianglizhou
Contributor Erik Joelsson <erikj@openjdk.org> successfully added.

[erikj79]

We also need to add some things to VARDEPS. Looks like at least $$($1_LD) and $$($1_SYSROOT_LDFLAGS) are needed. Maybe conditionally.

[jianglizhou]

Fix, thanks.

[jianglizhou]

Hi @erikj79 Please let me know if you have any additional comments for the PR. Could you please also help run through testing for the latest version? Thanks

[erikj79]

Hi @erikj79 Please let me know if you have any additional comments for the PR. Could you please also help run through testing for the latest version? Thanks

I'm running a full set of builds now, but in my initial local build on mac, I see a lot of warnings like this:

.../Xcode/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/lib/libc++.tbd, ignoring unexpected dylib text stub file

Any idea what that could be caused by or if it's possible to eliminate?

[erikj79]

My build job is still running, but it has failed in two distinct ways already. See below for mac fix. Our cross build of arm32 fails with this message:

[2023-05-24T19:25:15,310Z] /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/bin/ld: fatal error: cannot mix -r with dynamic object /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/lib/libstdc++.so

[jianglizhou]

Hi @erikj79 Please let me know if you have any additional comments for the PR. Could you please also help run through testing for the latest version? Thanks

I'm running a full set of builds now, but in my initial local build on mac, I see a lot of warnings like this:

.../Xcode/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/lib/libc++.tbd, ignoring unexpected dylib text stub file

Any idea what that could be caused by or if it's possible to eliminate?

I observed the same warnings on my macosx-x86_64 local build. It's from the partial linking step. Not much information yielded when I searched for ignoring unexpected dylib text stub file. Search found https://opensource.apple.com/source/ld64/ld64-609/src/ld/InputFiles.cpp with the linker code throwing the warning (dylibsNotAllowed is true if _options.outputKind is Options::kObjectFile, which is related to what we are observing).

I tried linking a javastatic executable using the static libraries created with partial linking on macosx-x86_64 last week. I was able to execute javastatic in my quick test but didn't go further for more verification, as the runtime didn't work with the static build yet.

Haven't found any information on how to suppress the warning either.

[jianglizhou]

My build job is still running, but it has failed in two distinct ways already. See below for mac fix. Our cross build of arm32 fails with this message:

[2023-05-24T19:25:15,310Z] /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/bin/ld: fatal error: cannot mix -r with dynamic object /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/lib/libstdc++.so

If this is a problem with your partial link solution, then perhaps just employing the relative path trick for the ar command line on mac could be enough to handle long paths and lots of object files? That appears to be how we handle it with clang for linking already due to @-file problems.

The partial linking was originally suggested by C++/Clang toolchain folks to mitigate linker overhead that was observed during final executable link time. For a static library containing any object file (.o) that was compiled with ThinLTO (https://clang.llvm.org/docs/ThinLTO.html) enabled, linking an executable using the static library without distributed ThinLTO could experience more overhead and slow down linking. Solving the macosx ar limitation is a side-effect/benefit of using partial linking. We probably would want to include the partial linking even without the ar limitation.

Is it possible $$($1_SYSROOT_LDFLAGS) pulled in libstdc++.so as part of the input for partial linking with the linux-aarch64 cross build?

[jianglizhou]

My build job is still running, but it has failed in two distinct ways already. See below for mac fix. Our cross build of arm32 fails with this message:

[2023-05-24T19:25:15,310Z] /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/bin/ld: fatal error: cannot mix -r with dynamic object /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/lib/libstdc++.so

If this is a problem with your partial link solution, then perhaps just employing the relative path trick for the ar command line on mac could be enough to handle long paths and lots of object files? That appears to be how we handle it with clang for linking already due to @-file problems.

The partial linking was originally suggested by C++/Clang toolchain folks to mitigate linker overhead that was observed during final executable link time. For a static library containing any object file (.o) that was compiled with ThinLTO (https://clang.llvm.org/docs/ThinLTO.html) enabled, linking an executable using the static library without distributed ThinLTO could experience more overhead and slow down linking. Solving the macosx ar limitation is a side-effect/benefit of using partial linking. We probably would want to include the partial linking even without the ar limitation.

Is it possible $$($1_SYSROOT_LDFLAGS) pulled in libstdc++.so as part of the input for partial linking with the linux-aarch64 cross build?

Another possibility might be the user provided BUILD_LDCXX includes extra options in the testing build (?). If that's the case, we probably could define a separate BUILD_LD_PARTIAL with no added options. In our prototype on JDK 11, we define a separate one for partial linking.

[jianglizhou]

My build job is still running, but it has failed in two distinct ways already. See below for mac fix. Our cross build of arm32 fails with this message:

[2023-05-24T19:25:15,310Z] /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/bin/ld: fatal error: cannot mix -r with dynamic object /opt/mach5/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/lib/libstdc++.so

If this is a problem with your partial link solution, then perhaps just employing the relative path trick for the ar command line on mac could be enough to handle long paths and lots of object files? That appears to be how we handle it with clang for linking already due to @-file problems.

The partial linking was originally suggested by C++/Clang toolchain folks to mitigate linker overhead that was observed during final executable link time. For a static library containing any object file (.o) that was compiled with ThinLTO (https://clang.llvm.org/docs/ThinLTO.html) enabled, linking an executable using the static library without distributed ThinLTO could experience more overhead and slow down linking. Solving the macosx ar limitation is a side-effect/benefit of using partial linking. We probably would want to include the partial linking even without the ar limitation.
Is it possible $$($1_SYSROOT_LDFLAGS) pulled in libstdc++.so as part of the input for partial linking with the linux-aarch64 cross build?

Another possibility might be the user provided BUILD_LDCXX includes extra options in the testing build (?). If that's the case, we probably could define a separate BUILD_LD_PARTIAL with no added options. In our prototype on JDK 11, we define a separate one for partial linking.

I tried building static-libs-image for linux-aarch64 on Ubuntu machine using devkit to reproduce the failure. Also tried building for linux-ppc64le-server-release target using devkit on Ubuntu. Both built successfully with using devkit. I could not build a devkit for arm32 (with make TARGETS="arm-linux-gnueabihf" BASE_OS=Fedora BASE_OS_VERSION=17, also tried with BASE_OS_VERSION=17).

@erikj79 Could you please help provide additional insight for the build failure you found for arm32? BUILD_LIBJVM_partial_link.cmdline might help shed some light.

For the aarch64-linux-gnu build using devkit, I see following, which is ok. No unexpected options are included.

/.../bin/aarch64-linux-gnu-g++ -r --sysroot=/.../aarch64-linux-gnu/sysroot -o /...linux-aarch64-server-release/hotspot/variant-server/libjvm/objs/static/libjvm_relocatable.o @/.../hotspot/variant-server/libjvm/objs/static/_BUILD_LIBJVM_objectfilenames.txt

[jianglizhou]

with make TARGETS="arm-linux-gnueabihf" BASE_OS=Fedora BASE_OS_VERSION=17, also tried with BASE_OS_VERSION=17

Fix typo: also tried with BASE_OS_VERSION=18

[erikj79]

Another possibility might be the user provided BUILD_LDCXX includes extra options in the testing build (?). If that's the case, we probably could define a separate BUILD_LD_PARTIAL with no added options. In our prototype on JDK 11, we define a separate one for partial linking.

I tried building static-libs-image for linux-aarch64 on Ubuntu machine using devkit to reproduce the failure. Also tried building for linux-ppc64le-server-release target using devkit on Ubuntu. Both built successfully with using devkit. I could not build a devkit for arm32 (with make TARGETS="arm-linux-gnueabihf" BASE_OS=Fedora BASE_OS_VERSION=17, also tried with BASE_OS_VERSION=17).

@erikj79 Could you please help provide additional insight for the build failure you found for arm32? BUILD_LIBJVM_partial_link.cmdline might help shed some light.

There are no extra options. I suspect it's an issue with the older GCC version. Here is one failing command line:

$ ( cd /home/erik/git/jdk/build/linux-arm32 ; /var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -r --sysroot=/var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/arm-linux-gnueabihf/sysroot -o /home/erik/git/jdk/build/linux-arm32/support/native/java.rmi/librmi/static/librmi_relocatable.o /home/erik/git/jdk/build/linux-arm32/support/native/java.rmi/librmi/static/GC.o )
/var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/bin/ld: fatal error: cannot mix -r with dynamic object /var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/lib/libgcc_s.so.1

The partial linking was originally suggested by C++/Clang toolchain folks to mitigate linker overhead that was observed during final executable link time. For a static library containing any object file (.o) that was compiled with ThinLTO (https://clang.llvm.org/docs/ThinLTO.html) enabled, linking an executable using the static library without distributed ThinLTO could experience more overhead and slow down linking. Solving the macosx ar limitation is a side-effect/benefit of using partial linking. We probably would want to include the partial linking even without the ar limitation.

This seems to be Clang specific, so maybe only apply this to Clang and not GCC? Have you measured the difference in link time and concluded that this workaround is needed? If so, how big were the differences? Given that it prints a lot of warnings on mac, I would suggest leaving the partial linking out of this patch to get something in that is actually working. It was included in this patch because of the side effect it had with handling long path names. However, since looking closer at that issue, we were using a different workaround for Clang already and that same workaround should work fine for AR on mac as well.

[jianglizhou]

Another possibility might be the user provided BUILD_LDCXX includes extra options in the testing build (?). If that's the case, we probably could define a separate BUILD_LD_PARTIAL with no added options. In our prototype on JDK 11, we define a separate one for partial linking.

I tried building static-libs-image for linux-aarch64 on Ubuntu machine using devkit to reproduce the failure. Also tried building for linux-ppc64le-server-release target using devkit on Ubuntu. Both built successfully with using devkit. I could not build a devkit for arm32 (with make TARGETS="arm-linux-gnueabihf" BASE_OS=Fedora BASE_OS_VERSION=17, also tried with BASE_OS_VERSION=17).
@erikj79 Could you please help provide additional insight for the build failure you found for arm32? BUILD_LIBJVM_partial_link.cmdline might help shed some light.

There are no extra options. I suspect it's an issue with the older GCC version. Here is one failing command line:

$ ( cd /home/erik/git/jdk/build/linux-arm32 ; /var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc -r --sysroot=/var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/arm-linux-gnueabihf/sysroot -o /home/erik/git/jdk/build/linux-arm32/support/native/java.rmi/librmi/static/librmi_relocatable.o /home/erik/git/jdk/build/linux-arm32/support/native/java.rmi/librmi/static/GC.o )
/var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/bin/ld: fatal error: cannot mix -r with dynamic object /var/tmp/jib-erik/install/jpg/infra/builddeps/devkit-linux_x64-to-linux_arm/gcc8.2.0-Fedora27+1.0/devkit-linux_x64-to-linux_arm-gcc8.2.0-Fedora27+1.0.tar.gz/x86_64-linux-gnu-to-arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/8.2.0/../../../../arm-linux-gnueabihf/lib/libgcc_s.so.1

Thanks a lot for looking into the issue.

The partial linking was originally suggested by C++/Clang toolchain folks to mitigate linker overhead that was observed during final executable link time. For a static library containing any object file (.o) that was compiled with ThinLTO (https://clang.llvm.org/docs/ThinLTO.html) enabled, linking an executable using the static library without distributed ThinLTO could experience more overhead and slow down linking. Solving the macosx ar limitation is a side-effect/benefit of using partial linking. We probably would want to include the partial linking even without the ar limitation.

This seems to be Clang specific, so maybe only apply this to Clang and not GCC? Have you measured the difference in link time and concluded that this workaround is needed? If so, how big were the differences?

It was noticeably slower (which prompted me looked into it at that time) when linking the executable using the static libraries created without partial linking workaround. In my experiments, it took >2m when linking final executable. With the mitigation the final linking took a few seconds.

Given that it prints a lot of warnings on mac, I would suggest leaving the partial linking out of this patch to get something in that is actually working. It was included in this patch because of the side effect it had with handling long path names. However, since looking closer at that issue, we were using a different workaround for Clang already and that same workaround should work fine for AR on mac as well.

Sounds good for solving the macosx ar limitation differently. I'll change that. We can also exclude the partial linking part for gcc (due the older tool issue that you've found). Any concerns with including partial linking step for clang on Linux?

[erikj79]

Sounds good for solving the macosx ar limitation differently. I'll change that. We can also exclude the partial linking part for gcc (due the older tool issue that you've found). Any concerns with including partial linking step for clang on Linux?

I have no concerns regarding clang on Linux.

[jianglizhou]

Sounds good for solving the macosx ar limitation differently. I'll change that. We can also exclude the partial linking part for gcc (due the older tool issue that you've found). Any concerns with including partial linking step for clang on Linux?

I have no concerns regarding clang on Linux.

I've updated this PR to include partial linking for building static libraries with clang on Linux only. On macosx builds, relative path is used when needed for long path and many objects. @erikj79, could you please help run another iteration of your tests to see if there are any remaining missed cases. Hopefully we'll be able to wrap up this change. Thanks!

[erikj79]

All our builds succeeded, so this is looking pretty good now. Just a minor suggestion left.

[erikj79]

This combination of conditions is repeated 3 times. Maybe we could assign the result to a variable (e.g. $1_ENABLE_PARTIAL_LINKING) and check for that instead?

[jianglizhou]

This combination of conditions is repeated 3 times. Maybe we could assign the result to a variable (e.g. $1_ENABLE_PARTIAL_LINKING) and check for that instead?

Done, thanks.

[jianglizhou]

All our builds succeeded, so this is looking pretty good now. Just a minor suggestion left.

Thanks a lot!

[openjdk]

@jianglizhou This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8307858: [REDO] JDK-8307194 Add make target for optionally building a complete set of all JDK and hotspot libjvm static libraries

Co-authored-by: Erik Joelsson <erikj@openjdk.org>
Reviewed-by: erikj

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been no new commits pushed to the master branch. If another commit should be pushed before you perform the /integrate command, your PR will be automatically rebased. If you prefer to avoid any potential automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[jianglizhou]

/integrate

[openjdk]

Going to push as commit 45414fc.
Since your change was applied there have been 15 commits pushed to the master branch:

• a6ad42e: 8308645: Javadoc of FFM API needs to be refreshed
• 0727507: 8272147: Consolidate preserved marks handling with other STW collectors
• 6cd166a: 8309403: Serial: Remove the useless adaptive size policy in GenCollectedHeap
• 5d5ae35: 8308966: Add intrinsic for float/double modulo for x86 AVX2 and AVX512
• 8e4e6b0: 8309761: Leak class loader constraints
• fdaa2c4: 8309306: G1: Move is_obj_dead from HeapRegion to G1CollectedHeap
• 4bc6bbb: 8309814: [IR Framework] Dump socket output string in which IR encoding was not found
• cf9e635: 8309462: [AIX] vmTestbase/nsk/jvmti/RunAgentThread/agentthr001/TestDescription.java crashing due to empty while loop
• 268ec61: 8308184: Launching java with large number of jars in classpath with java.protocol.handler.pkgs system property set can lead to StackOverflowError
• 4d47069: 4516654: Metalworks Demo: Window title not displayed fully in Low Vision Theme
• ... and 5 more: https://git.openjdk.org/jdk/compare/16c3d53b1bb60b1c1570731041f564bf13b45098...master

Your commit was automatically rebased without conflicts.

[openjdk]

@jianglizhou Pushed as commit 45414fc.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.