8264008: Incorrect metaspace statistics after JEP 387 when UseCompressedClassPointers is off #3142

DamonFool · 2021-03-23T03:50:36Z

Hi all,

Metaspace statistics is incorrect after JEP 387 when UseCompressedClassPointers is off.

For example, here is the incorrect metaspace statistics before the fix:

Event:jdk.MetaspaceSummary {
  startTime = 10:35:24.762
  gcId = 3
  when = "Before GC"
  gcThreshold = 21.0 MB
  metaspace = {
    committed = 10.3 MB
    used = 10.2 MB
    reserved = 16.0 MB
  }
  dataSpace = {
    committed = 10.3 MB
    used = 10.2 MB
    reserved = 16.0 MB
  }
  classSpace = {
    committed = 10.3 MB
    used = 10.2 MB
    reserved = 16.0 MB
  }
}

This bug can be reproduced by running the following tests with -XX:-UseCompressedClassPointers, which would pass before JEP 387.

jdk/jfr/event/gc/heapsummary/TestHeapSummaryEventDefNewSerial.java
jdk/jfr/event/gc/heapsummary/TestHeapSummaryEventG1.java
jdk/jfr/event/gc/heapsummary/TestHeapSummaryEventPSParOld.java

The failing reason is that Metaspace::is_class_space_allocation(mdtype) [1] will always return false when UseCompressedClassPointers is off.
So RunningCounters::committed_words_nonclass() will be returned even called with Metaspace::is_class_space_allocation(Metaspace::ClassType), which is unreasonable.
And MetaspaceUtils::reserved_words(Metaspace::MetadataType mdtype)/MetaspaceUtils::used_words(Metaspace::MetadataType mdtype) also suffer from the same bug.

Thanks.
Best regards,
Jie

[1] https://github.com/openjdk/jdk/blob/master/src/hotspot/share/memory/metaspace.cpp#L90

Metaspace statistics after the fix:

Event:jdk.MetaspaceSummary {
  startTime = 10:44:38.230
  gcId = 1
  when = "After GC"
  gcThreshold = 21.0 MB
  metaspace = {
    committed = 10.3 MB
    used = 10.2 MB
    reserved = 16.0 MB
  }
  dataSpace = {
    committed = 10.3 MB
    used = 10.2 MB
    reserved = 16.0 MB
  }
  classSpace = {
    committed = 0 bytes
    used = 0 bytes
    reserved = 0 bytes
  }
}

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8264008: Incorrect metaspace statistics after JEP 387 when UseCompressedClassPointers is off

Reviewers

Thomas Stuefe (@tstuefe - Reviewer)

Download

To checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3142/head:pull/3142
$ git checkout pull/3142

To update a local copy of the PR:
$ git checkout pull/3142
$ git pull https://git.openjdk.java.net/jdk pull/3142/head

…sedClassPointers is off

DamonFool · 2021-03-23T03:51:01Z

/issue add JDK-8264008
/test
/label add hotspot-runtime
/cc hotspot-runtime

bridgekeeper · 2021-03-23T03:51:21Z

👋 Welcome back jiefu! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-03-23T03:52:31Z

@DamonFool This issue is referenced in the PR title - it will now be updated.

openjdk · 2021-03-23T03:52:47Z

@DamonFool
The hotspot-runtime label was successfully added.

openjdk · 2021-03-23T03:53:35Z

@DamonFool The hotspot-runtime label was already applied.

mlbridge · 2021-03-23T03:55:50Z

Webrevs

tstuefe

Yes, this makes sense. Thanks for fixing this.

The problem is that is_class_space_allocation is ambiguous. The way it is implemented it answers the question "should this be allocated from class space". But in this statistics, it is used as a simple "is this a class space type".

What concerns me is that we did not find this earlier. Can you please add a test run for at least one of the failing tests to also execute with compressed oops off?

Even better would be an own regression test, e.g. as part of the gtests. Can be very simple, if UseCompressedClassPointers is off, MetaspaceUtils::committed_words should return 0 for class type. Then we can add another gtest run to the gtest runner jtreg tests with that option (possibly in a separate RFE).

DamonFool · 2021-03-23T07:45:12Z

Yes, this makes sense. Thanks for fixing this.

The problem is that is_class_space_allocation is ambiguous. The way it is implemented it answers the question "should this be allocated from class space". But in this statistics, it is used as a simple "is this a class space type".

What concerns me is that we did not find this earlier. Can you please add a test run for at least one of the failing tests to also execute with compressed oops off?

Even better would be an own regression test, e.g. as part of the gtests. Can be very simple, if UseCompressedClassPointers is off, MetaspaceUtils::committed_words should return 0 for class type. Then we can add another gtest run to the gtest runner jtreg tests with that option (possibly in a separate RFE).

Thanks @tstuefe for your review.

A test run has been added in jdk/jfr/event/gc/heapsummary/TestHeapSummaryEventDefNewSerial.java.
Will add more specific tests in a separate RFE.
Thanks.

tstuefe

Hi Jie,

thanks for adding the test. Notes inline.

Cheers, Thomas

tstuefe · 2021-03-23T16:51:02Z

test/jdk/jdk/jfr/event/gc/heapsummary/TestHeapSummaryEventDefNewSerial.java

+ * @requires vm.gc == "Serial" | vm.gc == null
+ * @library /test/lib /test/jdk
+ * @run main/othervm -XX:+IgnoreUnrecognizedVMOptions -XX:+UseSerialGC -XX:-UseCompressedClassPointers
+ *                   jdk.jfr.event.gc.heapsummary.TestHeapSummaryEventDefNewSerial


I would exclude 32bit here since it would just re-run the same test as above. Does the 32bit VM even accept UseCompressedClassPointers?

I would exclude 32bit here since it would just re-run the same test as above. Does the 32bit VM even accept UseCompressedClassPointers?

32bit has been excluded.
I don't think the 32bit VM accepts UseCompressedClassPointers and that's why there is -XX:+IgnoreUnrecognizedVMOptions in my previous patch.
Thanks.

tstuefe

Looks good!

openjdk · 2021-03-24T05:19:27Z

@DamonFool This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8264008: Incorrect metaspace statistics after JEP 387 when UseCompressedClassPointers is off

Reviewed-by: stuefe

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 34 new commits pushed to the master branch:

8d63bb6: 8260565: JFR: Fix copyright header in tests
0b2aa1b: 8263978: Clarify why 0 argument is ignored in SecureRandom::setSeed
15bcf6d: 8264055: backout JDK-8248904 in order to resubmit with additional attribution.
2425462: 8263903: Use Cleaner instead of finalize to auto stop Timer thread
35102cb: 8263992: Remove dead code NativeLookup::base_library_lookup
91d86e6: 8263572: Output from jstack mixed mode is misaligned
47ef038: 8263905: Remove finalize methods for SocketInput/OutputStream
1c9817b: 8261479: CDS runtime code should check exceptions
087c8bf: 8264041: Incorrect comments for ParallelCompactData::summarize_dense_prefix
c087f3e: 8263995: Incorrect double-checked locking in Types.arraySuperType()
... and 24 more: https://git.openjdk.java.net/jdk/compare/b23228d152ff8fa27bd32d9ef1307bf315039dea...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

DamonFool · 2021-03-24T09:30:25Z

Looks good!

Thanks @tstuefe .
/integrate

openjdk · 2021-03-24T09:31:32Z

@DamonFool Since your change was applied there have been 38 commits pushed to the master branch:

45e1bab: 8264091: Use the blessed modifier order in java.logging
cb776ed: 8263981: java.awt.image.ComponentSampleModel equals/hashcode use numBands twice
da512bf: 8264050: Remove unused field VM_HeapWalkOperation::_collecting_heap_roots
6e3a158: 8263352: assert(use == polladr) failed: the use should be a safepoint polling
8d63bb6: 8260565: JFR: Fix copyright header in tests
0b2aa1b: 8263978: Clarify why 0 argument is ignored in SecureRandom::setSeed
15bcf6d: 8264055: backout JDK-8248904 in order to resubmit with additional attribution.
2425462: 8263903: Use Cleaner instead of finalize to auto stop Timer thread
35102cb: 8263992: Remove dead code NativeLookup::base_library_lookup
91d86e6: 8263572: Output from jstack mixed mode is misaligned
... and 28 more: https://git.openjdk.java.net/jdk/compare/b23228d152ff8fa27bd32d9ef1307bf315039dea...master

Your commit was automatically rebased without conflicts.

Pushed as commit 06d46d6.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

DamonFool · 2021-04-09T12:37:16Z

Even better would be an own regression test, e.g. as part of the gtests. Can be very simple, if UseCompressedClassPointers is off, MetaspaceUtils::committed_words should return 0 for class type. Then we can add another gtest run to the gtest runner jtreg tests with that option (possibly in a separate RFE).

Hi @tstuefe ,

The gtest has been added here: #3414 .

Please review it.
Thanks.

8264008: Incorrect metaspace statistics after JEP 387 when UseCompres…

137b926

…sedClassPointers is off

openjdk bot added the rfr Pull request is ready for review label Mar 23, 2021

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Mar 23, 2021

tstuefe suggested changes Mar 23, 2021

View reviewed changes

Add a test run to reproduce the bug

c72e91e

tstuefe suggested changes Mar 23, 2021

View reviewed changes

Exclude 32-bit vm

10c9ab1

tstuefe approved these changes Mar 24, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Mar 24, 2021

openjdk bot closed this Mar 24, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Mar 24, 2021

DamonFool deleted the JDK-8264008 branch March 24, 2021 09:33

DamonFool mentioned this pull request Apr 9, 2021

8264983: Add gtest for JDK-8264008 #3414

Closed

3 tasks

8264008: Incorrect metaspace statistics after JEP 387 when UseCompressedClassPointers is off #3142

8264008: Incorrect metaspace statistics after JEP 387 when UseCompressedClassPointers is off #3142

Uh oh!

Conversation

DamonFool commented Mar 23, 2021 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Download

Uh oh!

DamonFool commented Mar 23, 2021

Uh oh!

bridgekeeper bot commented Mar 23, 2021

Uh oh!

openjdk bot commented Mar 23, 2021

Uh oh!

openjdk bot commented Mar 23, 2021

Uh oh!

openjdk bot commented Mar 23, 2021

Uh oh!

mlbridge bot commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

DamonFool commented Mar 23, 2021

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

tstuefe Mar 23, 2021

Choose a reason for hiding this comment

Uh oh!

DamonFool Mar 23, 2021

Choose a reason for hiding this comment

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Mar 24, 2021

Uh oh!

DamonFool commented Mar 24, 2021

Uh oh!

openjdk bot commented Mar 24, 2021

Uh oh!

DamonFool commented Apr 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

DamonFool commented Mar 23, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Mar 23, 2021 •

edited

Loading