Skip to content

8352064: AIX: now also able to build static-jdk image with a statically linked launcher#24062

Closed
JoKern65 wants to merge 5 commits intoopenjdk:masterfrom
JoKern65:JDK-8352064
Closed

8352064: AIX: now also able to build static-jdk image with a statically linked launcher#24062
JoKern65 wants to merge 5 commits intoopenjdk:masterfrom
JoKern65:JDK-8352064

Conversation

@JoKern65
Copy link
Contributor

@JoKern65 JoKern65 commented Mar 14, 2025

After "JDK-8339480: Build static-jdk image with a statically linked launcher" AIX was not able to build the new target. Therefore with "JDK-8345590 AIX 'make all' fails after JDK-8339480" the new target was disabled again.

Now with this change we can enable the statically linked launcher target again.
There are 3 things to do.

  1. Modify dladdr(). Because this API does not exist on AIX it is implemented based on the loadquery() API. Unfortunately, this API does only return the name of the executable, but not its path. Beforehand this was no problem, because we asked for a loaded library, for which the API provides the path. But now we are asking for the executable itself.
  2. dladdr() is differently implemented three times in the openjdk code. In the static case I supressed now the usage of the additional modules containing version two and three. I know this shouldn't be the final version. Magnus mentioned that they have discussed from time to time to have a "basic JDK utility lib" that can be shared between hotspot and the JDK libraries. I think this is a good idea for the future, but far beyond the scope of this PR. The second best thing Magnus proposed is to only have the dladdr() functionality in Hotspot and then export it. Let's see how the community decides.
  3. Because we lack a linker flag like whole-archive, I had to force the export of all symbols by creating export files containing all symbols of the static libs. I introduced the new rule for the export file generation as "raw" make recipes. Magnus claimed to use the SetupExecute. Unfortunately I was not able to make it function. So I still have my "raw" solution in place, but my last try with SetupExecute as comment beneath. Help is welcome.

Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue

Issue

  • JDK-8352064: AIX: now also able to build static-jdk image with a statically linked launcher (Bug - P4)

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/24062/head:pull/24062
$ git checkout pull/24062

Update a local copy of the PR:
$ git checkout pull/24062
$ git pull https://git.openjdk.org/jdk.git pull/24062/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 24062

View PR using the GUI difftool:
$ git pr show -t 24062

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/24062.diff

Using Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented Mar 14, 2025

👋 Welcome back jkern! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

openjdk bot commented Mar 14, 2025

@JoKern65 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8352064: AIX: now also able to build static-jdk image with a statically linked launcher

Reviewed-by: mdoerr, mbaesken, ihse

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 89 new commits pushed to the master branch:

  • 02a4ce2: 8352147: G1: TestEagerReclaimHumongousRegionsClearMarkBits test takes very long
  • de58009: 8351468: C2: array fill optimization assigns wrong type to intrinsic call
  • a875733: 8352486: [ubsan] compilationMemoryStatistic.cpp:659:21: runtime error: index 64 out of bounds for type const struct unnamed struct
  • 5591f8a: 8351515: C2 incorrectly removes double negation for double and float
  • 56a4ffa: 8352597: [IR Framework] test bug: TestNotCompilable.java fails on product build
  • e23e0f8: 8352591: Missing UnlockDiagnosticVMOptions in VerifyGraphEdgesWithDeadCodeCheckFromSafepoints test
  • adfb120: 8351748: Add class init barrier to AOT-cached Method/Var Handles
  • ee1577b: 8352652: [BACKOUT] nsk/jvmti/ tests should fail when nsk_jvmti_setFailStatus() is called
  • df9210e: 8347706: jvmciEnv.cpp has jvmci includes out of order
  • 5dd0acb: 8352477: RISC-V: Print warnings when unsupported intrinsics are enabled
  • ... and 79 more: https://git.openjdk.org/jdk/compare/712a70c5c44ac1fe916ceb1fff854d689b79b126...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot changed the title JDK-8352064: AIX: now also able to build static-jdk image with a statically linked launcher 8352064: AIX: now also able to build static-jdk image with a statically linked launcher Mar 14, 2025
@openjdk openjdk bot added the rfr Pull request is ready for review label Mar 14, 2025
@openjdk
Copy link

openjdk bot commented Mar 14, 2025

@JoKern65 The following labels will be automatically applied to this pull request:

  • build
  • client
  • core-libs
  • hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added hotspot-runtime hotspot-runtime-dev@openjdk.org build build-dev@openjdk.org client client-libs-dev@openjdk.org core-libs core-libs-dev@openjdk.org labels Mar 14, 2025
@mlbridge
Copy link

mlbridge bot commented Mar 14, 2025

Webrevs

# INFO := Generating export list for static libs, \
# DEPS := $(STATIC_LIB_FILE), \
# OUTPUT_FILE := $(STATIC_LIB_EXPORT_FILES), \
# COMMAND := $(AR) $(ARFLAGS) -w $(patsubst %.exp, %, $@) | $(GREP) -v '^\.' | $(AWK) '{print $$1}' | sort -u >$@, \
Copy link
Member

@erikj79 erikj79 Mar 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with using SetupExecute here is using $@ variables in the command line. To get this to work you need to delay resolving such variables. It may work by adding an adequate number of extra $, but I'm not sure how many would be needed or if it would actually work. You could try this, but it's probably trickier to get right than just doubling:

Suggested change
# COMMAND := $(AR) $(ARFLAGS) -w $(patsubst %.exp, %, $@) | $(GREP) -v '^\.' | $(AWK) '{print $$1}' | sort -u >$@, \
# COMMAND := $(AR) $(ARFLAGS) -w $$(patsubst %.exp, %, $$@) | $(GREP) -v '^\.' | $(AWK) '{print $$$$1}' | sort -u >$$@, \

You are probably better off using the explicit output file variable $(STATIC_LIB_EXPORT_FILES).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Eric, this might help with the command, but my problem is the dependency. The rules to generate the export files is not called, because in the build output I cannot find any line 'Generating export list for static libs'. Can you help out here too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem seems to be that $(generate_export_list) is empty and I do not know why.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this variable is misspelled (missing "S").

DEPS := $(STATIC_LIB_FILE), \

Also note that by default log level INFO is not printed. You would need to run the build with make LOG=info to see the message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ups, thank you for giving me the 'S'. But nevertheless if I make with
make all LOG=info an $(info generate_export_list: $(generate_export_list)) still shows an empty list

Copy link
Contributor Author

@JoKern65 JoKern65 Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not get the SetupExecute to function, even if I strip it down to a simple example

$(eval $(call SetupExecute, generate_export_list, \
    INFO := Generating export list for static libs, \
    DEPS := /path/libjli.a, \
    OUTPUT_FILE := /path/libjli.a.exp, \
    COMMAND := $(AR) $(ARFLAGS) -w $$(patsubst %.exp, %, $$@) | $(GREP) -v '^\.' | $(AWK) '{print $$$$1}' | $(SORT) -u >$$@, \
))

$(java): $(STATIC_LIB_FILES) /path/libjli.a.exp

I still get
gmake[3]: *** No rule to make target '/path/libjli.a.exp', needed by '/path2/java'. Stop.
Why does make not recognize, that the rule is in my
call SetupExecute, generate_export_list,
macro?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is likely due to the missing subshell. There is an enhancement filed to fix this automatically but it has unfortunately not been prioritized: https://bugs.openjdk.org/browse/JDK-8233115

STATIC_LIBS := -Wl,-bexpfull $(STATIC_LIB_FILES) $(addprefix -Wl$(COMMA)-bE:, $(STATIC_LIB_EXPORT_FILES))
$(STATIC_LIB_EXPORT_FILES): $(STATIC_LIB_FILES)
$(AR) $(ARFLAGS) -w $(patsubst %.exp, %, $@) | $(GREP) -v '^\.' | $(AWK) '{print $$1}' | sort -u >$@
#$(eval $(call SetupExecute, generate_export_list, \
Copy link
Member

@magicus magicus Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are multiple issues at hand here.

  1. There is a current bug in SetupExecute that requires you to execute any command with redirection in a subshell, that is, you need to wrap the COMMAND in ( ... ).

  2. STATIC_LIB_EXPORT_FILES is not a single file, it is a list of file. So what you have written above is not a single recipe, it is a multi-file recipe template. But it's weird, it will evaluate to like:
    libjvm.so.exp libjava.so.exp libjli.so.exp: libjvm.so libjava.so libjli.so
    which makes no sense, since you do not need to have a dependency to libjava.so libjli.so for libjvm.so.exp.

What you really want is to have a single rule per library that creates the corresponding exp file for that library, and which is dependent only on that library.

Or, possibly, you can just concatenate the export of all libraries into a single file. It will make the final linker command line simpler, since you would only need a single -Wl,-bE:<the export file>, but it is potentially worse for an incremental rebuild, since all native libraries need to be re-scanned for exported functions.

My preferred solution would to make this explicit by creating a loop over STATIC_LIB_FILES, and doing a SetupExecute for each such lib. Then you can get it down to something like:

$(foreach lib, $(STATIC_LIB_FILES), \
  $(eval $(call SetupExecute, generate_export_list_$(lib),
      INFO := Generating export list for $(lib), \
      DEPS :=  $(lib), \
      OUTPUT_FILE := $(lib).exp, \
      COMMAND := ( $(AR) $(ARFLAGS) -w $(lib) | $(GREP) -v '^\.' | $(AWK) '{print $$1}' | sort -u > $(lib).exp ), \
  ) \
  $(eval STATIC_LIB_EXPORT_FILES += $(lib).exp) \
)

))

$(java): $(STATIC_LIB_FILES)
$(java): $(STATIC_LIB_FILES) $(if $(call isTargetOs, aix), $(STATIC_LIB_EXPORT_FILES))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just clunky. Leave the original line, and add:

ifeq ($(call isTargetOs, aix), true)
  $(java): $(STATIC_LIB_EXPORT_FILES)
endif

@JoKern65
Copy link
Contributor Author

I applied all of magnus proposals, but I still get
gmake[3]: *** No rule to make target '/static_libs/jdk/build/aix-ppc64-server-fastdebug/support/native/java.base/libverify/static/libverify.a.exp', needed by '/static_libs/jdk/build/aix-ppc64-server-fastdebug/support/static-native/launcher/java'. Stop.

@JoKern65
Copy link
Contributor Author

I found the problem:
include Execute.gmk
was missing. Now in the same directory beside the three files e.g.

libjli.a
BUILD_LIBJLI_run_ar.cmdline
BUILD_LIBJLI_run_ar

I also get

libjli.a.exp
_generate_export_list_libjli.a_exec.cmdline
_generate_export_list_libjli.a_exec

So I think point 3 of the PR description is solved, although I find the name _generate_export_list_libjli.a_exec a little bit to long.

The next point to work on is No. 2 of the PR description (3 dladdr() implementations spread over the code). Who can help me out with a review and a tip in which direction I should go?

@magicus
Copy link
Member

magicus commented Mar 21, 2025

Regaring point 2: the current solution works, and is acceptable, I think. The two excluded files only contain the dladdr() implementation, and this is implemented in hotspot, so no functionality is lost. Compacting this into just a single implementation is a separate issue, and need not be resolved now.

Regarding the name of _generate_export_list_libjli.a_exec: this is generated from the name of the SetupExecute rule, generate_export_list_$(lib). I don't mind the name, it is clear. I don't see any particular point in keeping the name short, but if it bothers you, shorten the name of the rule. Try keeping it clear what it does, though.

Comment on lines +109 to +117
STATIC_LIB_EXPORT_FILES := $(foreach lib, $(STATIC_LIB_FILES), $(lib).exp)
STATIC_LIBS := -Wl,-bexpfull $(STATIC_LIB_FILES) $(addprefix -Wl$(COMMA)-bE:, $(STATIC_LIB_EXPORT_FILES))
$(foreach lib, $(STATIC_LIB_FILES), \
$(eval $(call SetupExecute, generate_export_list_$(notdir $(lib)), \
INFO := Generating export list for $(notdir $(lib)), \
DEPS := $(lib), \
OUTPUT_FILE := $(lib).exp, \
COMMAND := ( $(AR) $(ARFLAGS) -w $(lib) | $(GREP) -v '^\.' | $(AWK) '{print $$1}' | $(SORT) -u > $(lib).exp ), \
)) \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To better align with how we store the targets of Setup calls, can you please change this to:

Suggested change
STATIC_LIB_EXPORT_FILES := $(foreach lib, $(STATIC_LIB_FILES), $(lib).exp)
STATIC_LIBS := -Wl,-bexpfull $(STATIC_LIB_FILES) $(addprefix -Wl$(COMMA)-bE:, $(STATIC_LIB_EXPORT_FILES))
$(foreach lib, $(STATIC_LIB_FILES), \
$(eval $(call SetupExecute, generate_export_list_$(notdir $(lib)), \
INFO := Generating export list for $(notdir $(lib)), \
DEPS := $(lib), \
OUTPUT_FILE := $(lib).exp, \
COMMAND := ( $(AR) $(ARFLAGS) -w $(lib) | $(GREP) -v '^\.' | $(AWK) '{print $$1}' | $(SORT) -u > $(lib).exp ), \
)) \
STATIC_LIBS := -Wl,-bexpfull $(STATIC_LIB_FILES) $(addprefix -Wl$(COMMA)-bE:, $(STATIC_LIB_EXPORT_FILES))
$(foreach lib, $(STATIC_LIB_FILES), \
$(eval $(call SetupExecute, generate_export_list_$(notdir $(lib)), \
INFO := Generating export list for $(notdir $(lib)), \
DEPS := $(lib), \
OUTPUT_FILE := $(lib).exp, \
COMMAND := ( $(AR) $(ARFLAGS) -w $(lib) | $(GREP) -v '^\.' | $(AWK) '{print $$1}' | $(SORT) -u > $(lib).exp ), \
)) \
$(eval STATIC_LIB_EXPORT_FILES += $(lib).exp) \

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Member

@magicus magicus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build changes look good to me now. You need to have someone review the hotspot AIX changes as well.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Mar 21, 2025
Copy link
Contributor

@TheRealMDoerr TheRealMDoerr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really familiar with it, but I have some code style requests.

if (pgmpath[0] == 0) {
procentry64 PInfo;
PInfo.pi_pid = ::getpid();
if ( 0 == ::getargs(&PInfo, sizeof(PInfo), (char*)pgmpath,PATH_MAX) && *pgmpath ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coding style: spaces after ( and before ) are uncommon, but we should have one before PATH_MAX.
I think hotspot style guide compliant would be:
if (0 == ::getargs(&PInfo, sizeof(PInfo), (char*)pgmpath, PATH_MAX) && *pgmpath != 0)

pgmpath[PATH_MAX] = '\0';
pgmbase = strrchr(pgmpath, '/');
if (pgmbase) {
pgmbase +=1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a space: "+= 1"?

if ( 0 == ::getargs(&PInfo, sizeof(PInfo), (char*)pgmpath,PATH_MAX) && *pgmpath ) {
pgmpath[PATH_MAX] = '\0';
pgmbase = strrchr(pgmpath, '/');
if (pgmbase) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better: if (pgmbase != nullptr)

lm->data_len = ldi->ldinfo_datasize;

lm->path = g_stringlist.add(ldi->ldinfo_filename);
if (pgmbase && 0 == strcmp(pgmbase, ldi->ldinfo_filename)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pmgbase != nullptr

@openjdk openjdk bot removed the ready Pull request is ready to be integrated label Mar 21, 2025
Copy link
Contributor

@TheRealMDoerr TheRealMDoerr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Maybe @MBaesken wants to take a look at the AIX code, too?

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Mar 21, 2025
@MBaesken
Copy link
Member

Regarding BitmapToYXBandedRectangles from splashscreen, couldn't we just rename it to avoid clashes/issues with the other one from awt with the same name but other signature?
But this can be done in another issue/PR not here.

lm->data_len = ldi->ldinfo_datasize;

lm->path = g_stringlist.add(ldi->ldinfo_filename);
if (pgmbase != nullptr && 0 == strcmp(pgmbase, ldi->ldinfo_filename)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better strcmp(...) == 0 this is what we use almost everywhere in the HS codebase .

Copy link
Member

@MBaesken MBaesken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay, small nit inside. I would prefer more self explaining variable names like program_full_path , program_base_name but up to you!

@magicus
Copy link
Member

magicus commented Mar 24, 2025

Regarding BitmapToYXBandedRectangles from splashscreen, couldn't we just rename it to avoid clashes/issues with the other one from awt with the same name but other signature?
But this can be done in another issue/PR not here.

Indeed we could. Note that the same issue apply to all unix platforms. It's sort of a hack -- the core problem is that we cannot hide non-exported symbols in static libraries. Solving that is the long-term goal, but it is tricky on several platforms.

It could make sense to try to unify the two versions of BitmapToYXBandedRectangles as well; they are similar but not identical (but I can't really tell why).

@MBaesken
Copy link
Member

Seems we use already a bit of functionality from common/awt in the splashscreen lib

140 ifeq ($(ENABLE_HEADLESS_ONLY), false)
141   LIBSPLASHSCREEN_EXTRA_SRC := \
142       common/awt/systemscale \
143       #
144 
145   LIBSPLASHSCREEN_HEADER_DIRS := \
146       common/awt/utility \

maybe it would make sense to centralize the BitmapToYXBandedRectangles there and reuse it from there?

@JoKern65
Copy link
Contributor Author

/integrate

@openjdk
Copy link

openjdk bot commented Mar 24, 2025

Going to push as commit d8c2f59.
Since your change was applied there have been 90 commits pushed to the master branch:

  • 5709f79: 8352648: JFR: 'jfr query' should not be available in product builds
  • 02a4ce2: 8352147: G1: TestEagerReclaimHumongousRegionsClearMarkBits test takes very long
  • de58009: 8351468: C2: array fill optimization assigns wrong type to intrinsic call
  • a875733: 8352486: [ubsan] compilationMemoryStatistic.cpp:659:21: runtime error: index 64 out of bounds for type const struct unnamed struct
  • 5591f8a: 8351515: C2 incorrectly removes double negation for double and float
  • 56a4ffa: 8352597: [IR Framework] test bug: TestNotCompilable.java fails on product build
  • e23e0f8: 8352591: Missing UnlockDiagnosticVMOptions in VerifyGraphEdgesWithDeadCodeCheckFromSafepoints test
  • adfb120: 8351748: Add class init barrier to AOT-cached Method/Var Handles
  • ee1577b: 8352652: [BACKOUT] nsk/jvmti/ tests should fail when nsk_jvmti_setFailStatus() is called
  • df9210e: 8347706: jvmciEnv.cpp has jvmci includes out of order
  • ... and 80 more: https://git.openjdk.org/jdk/compare/712a70c5c44ac1fe916ceb1fff854d689b79b126...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Mar 24, 2025
@openjdk openjdk bot closed this Mar 24, 2025
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Mar 24, 2025
@openjdk
Copy link

openjdk bot commented Mar 24, 2025

@JoKern65 Pushed as commit d8c2f59.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@magicus
Copy link
Member

magicus commented Mar 24, 2025

Seems we use already a bit of functionality from common/awt in the splashscreen lib

Maybe. The client team has traditionally been very conservative about doing clean-up changes like that, so you might be entering an uphill battle if you want to do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build build-dev@openjdk.org client client-libs-dev@openjdk.org core-libs core-libs-dev@openjdk.org hotspot-runtime hotspot-runtime-dev@openjdk.org integrated Pull request has been integrated

Development

Successfully merging this pull request may close these issues.

5 participants