8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS #2275

dholmes-ora · 2021-01-28T02:40:08Z

A simple but long standing bug whereby the MemoryPool MXBean only sees the value of MaxMetaspaceSize if it was set directly on the command-line and not by other means (e.g. env var).

Fix is to check !FLAG_IS_DEFAULT rather than FLAG_IS_CMDLINE.

Expanded regression test added (based on the reproducer in the JBS issue) that checks all the ways to set the flag: cmd-line, env-var, hotspotrc file

Testing: regression test (before and after fix)
tiers 1-3 (in progress)

Thanks,
David

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS

Reviewers

Aleksey Shipilev (@shipilev - Reviewer) ⚠️ Review applies to 8f3d989
Thomas Stuefe (@tstuefe - Reviewer) ⚠️ Review applies to 2c3cad6

Download

$ git fetch https://git.openjdk.java.net/jdk pull/2275/head:pull/2275
$ git checkout pull/2275

…TOOLS_OPTIONS

bridgekeeper · 2021-01-28T02:40:35Z

👋 Welcome back dholmes! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-01-28T02:41:41Z

@dholmes-ora The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-01-28T02:45:57Z

Webrevs

shipilev

The fix looks right. I suggest polishing the test a little.

shipilev · 2021-01-28T09:57:59Z

test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeEnvVarTest.java

+import java.io.PrintWriter;
+import java.lang.management.ManagementFactory;
+import java.lang.management.MemoryPoolMXBean;
+
+import jdk.test.lib.process.ProcessTools;
+import jdk.test.lib.process.OutputAnalyzer;
+
+/*
+ * @test
+ * @bug 8260349
+ * @summary test that setting via the env-var and options file shows up as expected
+ * @library /test/lib
+ * @run driver MaxMetaspaceSizeEnvVarTest
+ */


I believe the conventional style is to do jtreg test definition the first thing, and then do imports.

Thanks for the Review Aleksey - note I just updated the test with a missing test case.

I'm not aware of a specific convention here. :) I used the existing MaxMetaspaceSizeTest.java in the same directory as the template for this one. But I can change it if you insist.

Thanks,
David

Please change if you are doing the test touchups anyway.

test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeEnvVarTest.java

openjdk · 2021-01-28T10:04:06Z

@dholmes-ora This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS

Reviewed-by: shade, stuefe

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been no new commits pushed to the master branch. If another commit should be pushed before you perform the /integrate command, your PR will be automatically rebased. If you prefer to avoid any potential automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

shipilev

Good. Minor nits to fix before push?

test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeEnvVarTest.java

tstuefe

Hi David,

this looks fine to me. Some nits below, but feel free to ignore them. Thanks for fixing this.

..Thomas

tstuefe · 2021-01-28T10:52:41Z

src/hotspot/share/services/memoryPool.cpp

-  return FLAG_IS_CMDLINE(MaxMetaspaceSize) ? MaxMetaspaceSize :
-                                             MemoryUsage::undefined_size();
+  return !FLAG_IS_DEFAULT(MaxMetaspaceSize) ? MaxMetaspaceSize :
+                                              MemoryUsage::undefined_size();


Somewhat more robust may be to compare against max_uint. In case we ever change the default to not be infinite. But I am fine with this too.

tstuefe · 2021-01-28T11:01:14Z

test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeEnvVarTest.java

+    }
+
+    public static void main(String... args) throws Exception {
+        final String max = String.valueOf(9 * 1024 * 1024); // 9 MB


You could set this to a much larger value for this test, since it won't change how the test works and would not cause more footprint. Does not matter here but may matter if we ever backport this, since before jep387 initial metaspace usage was a lot higher (400K vs 5-6m).

mlbridge · 2021-01-28T13:16:44Z

Mailing list message from David Holmes on hotspot-runtime-dev:

Hi Thomas,

Thanks for taking a look at this.

On 28/01/2021 9:33 pm, Thomas Stuefe wrote:

On Thu, 28 Jan 2021 10:53:02 GMT, David Holmes <dholmes at openjdk.org> wrote:

A simple but long standing bug whereby the MemoryPool MXBean only sees the value of MaxMetaspaceSize if it was set directly on the command-line and not by other means (e.g. env var).

Fix is to check !FLAG_IS_DEFAULT rather than FLAG_IS_CMDLINE.

Expanded regression test added (based on the reproducer in the JBS issue) that checks all the ways to set the flag: cmd-line, env-var, hotspotrc file

Testing: regression test (before and after fix)
tiers 1-3 (in progress)

Thanks,
David

David Holmes has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains seven additional commits since the last revision:

- Further formatting updates from Aleksey
- Merge branch 'master' into 8260349
- Cleaned up test formatting based on Aleksey's review
- Added missing test case for flag not set
- Boost max from 1MB to 9MB as 1MB was too low for Aarch64 to startup
- Reduced test code using loop for env-vars
- 8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOLS_OPTIONS

Hi David,

this looks fine to me. Some nits below, but feel free to ignore them. Thanks for fixing this.

..Thomas

src/hotspot/share/services/memoryPool.cpp line 201:

199: size_t MetaspacePool::calculate_max_size() const {
200: return !FLAG_IS_DEFAULT(MaxMetaspaceSize) ? MaxMetaspaceSize :
201: MemoryUsage::undefined_size();

Somewhat more robust may be to compare against max_uint. In case we ever change the default to not be infinite. But I am fine with this too.

I'll leave it as is.

test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeEnvVarTest.java line 66:

64:
65: public static void main(String... args) throws Exception {
66: final String max = String.valueOf(9 * 1024 * 1024); // 9 MB

You could set this to a much larger value for this test, since it won't change how the test works and would not cause more footprint. Does not matter here but may matter if we ever backport this, since before jep387 initial metaspace usage was a lot higher (400K vs 5-6m).

Actually the 9MB was selected because I found that on earlier versions
it was a lot higher. 8M failed and 9M passed.

Thanks again.
David
-----

dholmes-ora · 2021-02-01T21:30:58Z

/integrate

openjdk · 2021-02-01T21:31:48Z

@dholmes-ora Pushed as commit b6a7367.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

dholmes-ora added 2 commits January 27, 2021 21:19

8260349: Cannot programmatically retrieve Metaspace max set via JAVA_…

5087c14

…TOOLS_OPTIONS

Reduced test code using loop for env-vars

cf88770

dholmes-ora marked this pull request as ready for review January 28, 2021 02:40

openjdk bot added the rfr Pull request is ready for review label Jan 28, 2021

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Jan 28, 2021

dholmes-ora changed the title ~~8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOLS_OPTIONS~~ 8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS Jan 28, 2021

Boost max from 1MB to 9MB as 1MB was too low for Aarch64 to startup

8c814dc

shipilev approved these changes Jan 28, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jan 28, 2021

dholmes-ora added 3 commits January 28, 2021 05:30

Added missing test case for flag not set

a21b75b

Cleaned up test formatting based on Aleksey's review

8f3d989

Merge branch 'master' into 8260349

c8e5541

shipilev approved these changes Jan 28, 2021

View reviewed changes

test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeEnvVarTest.java Outdated Show resolved Hide resolved

test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeEnvVarTest.java Outdated Show resolved Hide resolved

dholmes-ora added 2 commits January 28, 2021 05:49

Further formatting updates from Aleksey

2c3cad6

Forgot to move imports.

14f6025

tstuefe approved these changes Jan 28, 2021

View reviewed changes

Merge branch 'master' into 8260349

cfe6587

openjdk bot closed this Feb 1, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Feb 1, 2021

dholmes-ora deleted the 8260349 branch February 1, 2021 21:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS #2275

8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS #2275

dholmes-ora commented Jan 28, 2021 •

edited by openjdk bot

Loading

bridgekeeper bot commented Jan 28, 2021

openjdk bot commented Jan 28, 2021

mlbridge bot commented Jan 28, 2021 •

edited

Loading

shipilev left a comment

shipilev Jan 28, 2021

dholmes-ora Jan 28, 2021

shipilev Jan 28, 2021

openjdk bot commented Jan 28, 2021 •

edited

Loading

shipilev left a comment

tstuefe left a comment

tstuefe Jan 28, 2021

tstuefe Jan 28, 2021

mlbridge bot commented Jan 28, 2021

dholmes-ora commented Feb 1, 2021

openjdk bot commented Feb 1, 2021

8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS #2275

8260349: Cannot programmatically retrieve Metaspace max set via JAVA_TOOL_OPTIONS #2275

Conversation

dholmes-ora commented Jan 28, 2021 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Download

bridgekeeper bot commented Jan 28, 2021

openjdk bot commented Jan 28, 2021

mlbridge bot commented Jan 28, 2021 • edited Loading

Webrevs

shipilev left a comment

Choose a reason for hiding this comment

shipilev Jan 28, 2021

Choose a reason for hiding this comment

dholmes-ora Jan 28, 2021

Choose a reason for hiding this comment

shipilev Jan 28, 2021

Choose a reason for hiding this comment

openjdk bot commented Jan 28, 2021 • edited Loading

shipilev left a comment

Choose a reason for hiding this comment

tstuefe left a comment

Choose a reason for hiding this comment

tstuefe Jan 28, 2021

Choose a reason for hiding this comment

tstuefe Jan 28, 2021

Choose a reason for hiding this comment

mlbridge bot commented Jan 28, 2021

dholmes-ora commented Feb 1, 2021

openjdk bot commented Feb 1, 2021

dholmes-ora commented Jan 28, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Jan 28, 2021 •

edited

Loading

openjdk bot commented Jan 28, 2021 •

edited

Loading