JDK-8323964: runtime/Thread/ThreadCountLimit.java fails intermittently on AIX #17512

MBaesken · 2024-01-22T08:20:51Z

Every 1-2 months we run into into such kind of failures on AIX:

[2.498s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (11=EAGAIN) for attributes: stacksize: 2112k, guardsize: 0k, detached.
[2.498s][warning][os,thread] Number of threads approx. running in the VM: 4223
[2.498s][warning][os,thread] Checking JVM parameter MaxExpectedDataSegmentSize (currently 8388608k) might be helpful
[2.498s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Thread-4213"

[error occurred during error reporting (), id 0xe0000000, Internal Error (/workspace/openjdk-jdk-dev-aix_ppc64-dbg/jdk/src/hotspot/os/aix/os_aix.cpp:2971)]
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 56 bytes. Error detail: AllocateHeap
[thread 3856 also had an error]
# An error report file with more information is saved as:
# /tmp/hs_err_pid13959464.log
[thread 76331 also had an error]

Seems the MaxExpectedDataSegmentSize (Maximum expected size of the data segment, AIX specific flag)
settings in in some cases too low for the test.
We experienced this a few times in scenarios where a lot of threads are created (and even warn about this, see os_aix.cpp).
So we should better run with a higher than default value of MaxExpectedDataSegmentSize for this test.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8323964: runtime/Thread/ThreadCountLimit.java fails intermittently on AIX (Bug - P4)

Reviewers

David Holmes (@dholmes-ora - Reviewer) ⚠️ Review applies to de4dc68a
Thomas Stuefe (@tstuefe - Reviewer) ⚠️ Review applies to de4dc68a

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/17512/head:pull/17512
$ git checkout pull/17512

Update a local copy of the PR:
$ git checkout pull/17512
$ git pull https://git.openjdk.org/jdk.git pull/17512/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 17512

View PR using the GUI difftool:
$ git pr show -t 17512

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/17512.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2024-01-22T08:22:16Z

👋 Welcome back mbaesken! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-01-22T08:24:23Z

@MBaesken The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2024-01-22T08:27:34Z

Webrevs

dholmes-ora

LGTM.

Thanks

dholmes-ora · 2024-01-23T02:08:21Z

src/hotspot/os/aix/globals_aix.hpp

+  /*  We need to leave "breathing space" for the data segment when             */   \
+  /*  placing the java heap. If the MaxExpectedDataSegmentSize setting         */   \
+  /*  is too small, we might run into resource issues creating many native     */   \
+  /*  threads, if it is too large, we reduce our chance of getting a low hea   */   \


typo: hea -> heap

Thanks for correcting the comment; it made no sense before

openjdk · 2024-01-23T02:10:37Z

@MBaesken This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8323964: runtime/Thread/ThreadCountLimit.java fails intermittently on AIX

Reviewed-by: dholmes, stuefe

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 57 new commits pushed to the master branch:

5a74c2a: 8323438: Enhance assertions for Windows sync API failures
52523d3: 8324050: Issue store-store barrier after re-materializing objects during deoptimization
df370d7: 8314329: AgeTable: add is_clear() & allocation spec, and relax assert to allow use of 0-index slot
0d8543d: 8324065: Daylight saving information for Africa/Casablanca are incorrect
c9cacfb: 8323657: Compilation of snippet results in VerifyError at runtime with --release 9 (and above)
bde650f: 8322282: Incorrect LoaderConstraintTable::add_entry after JDK-8298468
be943a9: 8321984: IGV: Upgrade to Netbeans Platform 20
d3b2ac1: 8314186: runtime/8176717/TestInheritFD.java failed with "Log file was leaked"
72f1990: 8323057: Recoverable errors may be reported before unrecoverable errors when annotation processing is skipped
c84af49: 8324129: C2: Remove some ttyLocker usages in preparation for JDK-8306767
... and 47 more: https://git.openjdk.org/jdk/compare/b89172149d6a900d11630a95be7278870421b435...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

tstuefe

Good.

tstuefe · 2024-01-23T07:01:07Z

src/hotspot/os/aix/globals_aix.hpp

+  /*  We need to leave "breathing space" for the data segment when             */   \
+  /*  placing the java heap. If the MaxExpectedDataSegmentSize setting         */   \
+  /*  is too small, we might run into resource issues creating many native     */   \
+  /*  threads, if it is too large, we reduce our chance of getting a low hea   */   \


Thanks for correcting the comment; it made no sense before

MBaesken · 2024-01-23T08:15:09Z

/integrate

openjdk · 2024-01-23T08:17:01Z

Going to push as commit 3696765.
Since your change was applied there have been 57 commits pushed to the master branch:

5a74c2a: 8323438: Enhance assertions for Windows sync API failures
52523d3: 8324050: Issue store-store barrier after re-materializing objects during deoptimization
df370d7: 8314329: AgeTable: add is_clear() & allocation spec, and relax assert to allow use of 0-index slot
0d8543d: 8324065: Daylight saving information for Africa/Casablanca are incorrect
c9cacfb: 8323657: Compilation of snippet results in VerifyError at runtime with --release 9 (and above)
bde650f: 8322282: Incorrect LoaderConstraintTable::add_entry after JDK-8298468
be943a9: 8321984: IGV: Upgrade to Netbeans Platform 20
d3b2ac1: 8314186: runtime/8176717/TestInheritFD.java failed with "Log file was leaked"
72f1990: 8323057: Recoverable errors may be reported before unrecoverable errors when annotation processing is skipped
c84af49: 8324129: C2: Remove some ttyLocker usages in preparation for JDK-8306767
... and 47 more: https://git.openjdk.org/jdk/compare/b89172149d6a900d11630a95be7278870421b435...master

Your commit was automatically rebased without conflicts.

openjdk · 2024-01-23T08:17:07Z

@MBaesken Pushed as commit 3696765.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

JDK-8323964

47243ea

openjdk bot added the rfr Pull request is ready for review label Jan 22, 2024

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Jan 22, 2024

adjust comment about MaxExpectedDataSegmentSize

de4dc68

dholmes-ora approved these changes Jan 23, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jan 23, 2024

tstuefe approved these changes Jan 23, 2024

View reviewed changes

fix typo

6b38ccf

openjdk bot added the integrated Pull request has been integrated label Jan 23, 2024

openjdk bot closed this Jan 23, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jan 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK-8323964: runtime/Thread/ThreadCountLimit.java fails intermittently on AIX #17512

JDK-8323964: runtime/Thread/ThreadCountLimit.java fails intermittently on AIX #17512

MBaesken commented Jan 22, 2024 •

edited by openjdk bot

Loading

bridgekeeper bot commented Jan 22, 2024

openjdk bot commented Jan 22, 2024

mlbridge bot commented Jan 22, 2024 •

edited

Loading

dholmes-ora left a comment

dholmes-ora Jan 23, 2024

tstuefe Jan 23, 2024

openjdk bot commented Jan 23, 2024 •

edited

Loading

tstuefe left a comment

tstuefe Jan 23, 2024

MBaesken commented Jan 23, 2024

openjdk bot commented Jan 23, 2024

openjdk bot commented Jan 23, 2024

JDK-8323964: runtime/Thread/ThreadCountLimit.java fails intermittently on AIX #17512

JDK-8323964: runtime/Thread/ThreadCountLimit.java fails intermittently on AIX #17512

Conversation

MBaesken commented Jan 22, 2024 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Jan 22, 2024

openjdk bot commented Jan 22, 2024

mlbridge bot commented Jan 22, 2024 • edited Loading

Webrevs

dholmes-ora left a comment

Choose a reason for hiding this comment

dholmes-ora Jan 23, 2024

Choose a reason for hiding this comment

tstuefe Jan 23, 2024

Choose a reason for hiding this comment

openjdk bot commented Jan 23, 2024 • edited Loading

tstuefe left a comment

Choose a reason for hiding this comment

tstuefe Jan 23, 2024

Choose a reason for hiding this comment

MBaesken commented Jan 23, 2024

openjdk bot commented Jan 23, 2024

openjdk bot commented Jan 23, 2024

MBaesken commented Jan 22, 2024 •

edited by openjdk bot

Loading

mlbridge bot commented Jan 22, 2024 •

edited

Loading

openjdk bot commented Jan 23, 2024 •

edited

Loading