8253079: DeterministicDump.java fails due to garbage in structure padding #267

iklam · 2020-09-20T05:37:33Z

(EDITED) In product builds, when PackageEntry and ModuleEntry objects are allocated, the memory is not zeroed. As a result, the structure padding slots (such as the 32-bits after BasicHashtableEntry::_hash) may contain garbage values that are different on every run of java -Xshare:dump. As a result, java -Xshare:dump cannot reproduce deterministic result.

The fix is to clear the memory for the newly allocated HashtableEntry objects when DumpSharedSpaces == true.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8253079: DeterministicDump.java fails due to garbage in structure padding

Reviewers

Yumin Qi (@yminqi - Reviewer) ⚠️ Review applies to 225c927
Jie Fu (@DamonFool - Committer)
Aleksey Shipilev (@shipilev - Reviewer)

Download

$ git fetch https://git.openjdk.java.net/jdk pull/267/head:pull/267
$ git checkout pull/267

…tructure padding

bridgekeeper · 2020-09-20T05:39:38Z

👋 Welcome back iklam! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2020-09-20T05:40:42Z

@iklam The following label will be automatically applied to this pull request: hotspot-runtime.

When this pull request is ready to be reviewed, an RFR email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label (add|remove) "label" command.

mlbridge · 2020-09-20T05:51:53Z

Webrevs

yminqi · 2020-09-20T22:43:43Z

The implementation is quite complex, every derivatives from BasicHashTableEntry need to have a copy_from function to avoid such problem. Maybe a brutal one to avoid such padding issue for all cases is fill the object allocated in AllocateHeap with \0?

…age in structure padding" This reverts commit d380e02.

…terministicDump-test-fails-product-build

iklam · 2020-09-21T05:23:27Z

The implementation is quite complex, every derivatives from BasicHashTableEntry need to have a copy_from function to avoid such problem. Maybe a brutal one to avoid such padding issue for all cases is fill the object allocated in AllocateHeap with \0?

You're right. My original analysis was wrong: set_hash() didn't write garbage into the padding. Instead, the garbage was there because AllocaHeap didn't initialize the new buffer in product builds.

I reverted the original fix. Instead, I added code to call memset() when allocating a new hashtable entry (but only when DumpSharedSpaces is true).

shipilev · 2020-09-21T05:48:14Z

src/hotspot/share/classfile/moduleEntry.cpp

@@ -454,6 +454,7 @@ void ModuleEntry::init_as_archived_entry() {
  if (_location != NULL) {
    _location = ArchiveBuilder::get_relocated_symbol(_location);
  }
+  JFR_ONLY(memset(trace_id_addr(), 0, sizeof(traceid)));


memset looks dodgy here. Maybe JFR_ONLY(set_trace_id(0))?

I removed these from my previous commit, because the trace_id seemed to be predictable and wouldn't cause the archive content to change. Anyway, I've added the code you suggested in commit 5706a82. That way, if trace_ids become unpredictable due to future JFR implementation changes, the CDS code won't be affected.

shipilev · 2020-09-21T05:48:14Z

src/hotspot/share/classfile/packageEntry.cpp

@@ -238,6 +238,7 @@ void PackageEntry::init_as_archived_entry() {
  _module = ModuleEntry::get_archived_entry(_module);
  _qualified_exports = (GrowableArray<ModuleEntry*>*)archived_qualified_exports;
  _defined_by_cds_in_class_path = 0;
+  JFR_ONLY(memset(trace_id_addr(), 0, sizeof(traceid)));


Ditto, JFR_ONLY(set_trace_id(0))?

DamonFool · 2020-09-21T07:10:41Z

It passed with this fix.
Looks good to me.
Thanks.

DamonFool · 2020-09-21T07:22:51Z

/reviewer jiefu

openjdk · 2020-09-21T07:23:27Z

@DamonFool Only the author (@iklam) is allowed to issue the reviewer command.

yminqi

looks good!

openjdk · 2020-09-21T16:43:45Z

@iklam This change now passes all automated pre-integration checks. In addition to the automated checks, the change must also fulfill all project specific requirements

After integration, the commit message will be:

8253079: DeterministicDump.java fails due to garbage in structure padding

Reviewed-by: minqi, jiefu, shade

If you would like to add a summary, use the /summary command.
To credit additional contributors, use the /contributor command.
To add additional solved issues, use the /issue command.

Since the source branch of this PR was last updated there have been 19 commits pushed to the master branch:

a4affd5: 8253412: Unsupported GC options passed in JAVA_TOOL_OPTIONS are silently ignored
96f722c: 8252114: Windows-AArch64: Enable and test ZGC and ShenandoahGC
f7b1ce4: 8253447: Remove buggy code introduced by 8249451
282b9dc: 8081833: Clean up JVMFlag getter/setter code
0e98fc1: 8253237: [REDO] Improve large object handling during evacuation
d1f9b8a: 8234808: jdb quoted option parsing broken
d8921ed: 8247281: migrate ObjectMonitor::_object to OopStorage
f800af9: 8241390: 'Deadlock' with VM_RedefineClasses::lock_classes()
2e30ff6: 8253311: Cleanup relocInfo constructors
43be5a3: 8230652: Improve verbose output
... and 9 more: https://git.openjdk.java.net/jdk/compare/388c8f2538744c0ef328861c04f6e251500e8adc...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid automatic rebasing, please merge master into your branch, and then specify the current head hash when integrating, like this: /integrate a4affd5b04ab9ab409871561bdfa7536f2c32e79.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

iklam · 2020-09-21T17:42:23Z

/reviewer jiefu

openjdk · 2020-09-21T17:42:46Z

@iklam Syntax: /reviewer (credit|remove) [@user | openjdk-user]+. For example:

/reviewer credit @openjdk-bot
/reviewer credit duke
/reviewer credit @user1 @user2

iklam · 2020-09-21T17:43:53Z

/reviewer credit jiefu

openjdk · 2020-09-21T17:44:47Z

@iklam
Reviewer jiefu successfully credited.

iklam · 2020-09-22T08:03:42Z

/integrate

openjdk · 2020-09-22T08:04:37Z

@iklam Since your change was applied there have been 19 commits pushed to the master branch:

a4affd5: 8253412: Unsupported GC options passed in JAVA_TOOL_OPTIONS are silently ignored
96f722c: 8252114: Windows-AArch64: Enable and test ZGC and ShenandoahGC
f7b1ce4: 8253447: Remove buggy code introduced by 8249451
282b9dc: 8081833: Clean up JVMFlag getter/setter code
0e98fc1: 8253237: [REDO] Improve large object handling during evacuation
d1f9b8a: 8234808: jdb quoted option parsing broken
d8921ed: 8247281: migrate ObjectMonitor::_object to OopStorage
f800af9: 8241390: 'Deadlock' with VM_RedefineClasses::lock_classes()
2e30ff6: 8253311: Cleanup relocInfo constructors
43be5a3: 8230652: Improve verbose output
... and 9 more: https://git.openjdk.java.net/jdk/compare/388c8f2538744c0ef328861c04f6e251500e8adc...master

Your commit was automatically rebased without conflicts.

Pushed as commit 284bbf0.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8253079: runtime/cds/DeterministicDump.java fails due to garbage in s…

d380e02

…tructure padding

openjdk bot added the hotspot-runtime label Sep 20, 2020

iklam changed the title ~~8253079: runtime/cds/DeterministicDump.java fails due to garbage in structure padding~~ 8253079: DeterministicDump.java fails due to garbage in structure padding Sep 20, 2020

iklam marked this pull request as ready for review Sep 20, 2020

openjdk bot added the rfr label Sep 20, 2020

iklam added 3 commits Sep 21, 2020

Revert "8253079: runtime/cds/DeterministicDump.java fails due to garb…

352be95

…age in structure padding" This reverts commit d380e02.

Merge branch 'master' of https://github.com/iklam/jdk into 8253079-De…

336b0d6

…terministicDump-test-fails-product-build

Zero out memory for newly allocated HashtableEntry

225c927

shipilev reviewed Sep 21, 2020

View changes

yminqi approved these changes Sep 21, 2020

View changes

openjdk bot added the ready label Sep 21, 2020

also reset trace_id

5706a82

DamonFool approved these changes Sep 21, 2020

View changes

shipilev approved these changes Sep 22, 2020

View changes

openjdk bot closed this Sep 22, 2020

openjdk bot added the integrated label Sep 22, 2020

openjdk bot removed ready rfr labels Sep 22, 2020

8253079: DeterministicDump.java fails due to garbage in structure padding #267

8253079: DeterministicDump.java fails due to garbage in structure padding #267

iklam commented Sep 20, 2020 •

edited by openjdk[bot] bot

bridgekeeper bot commented Sep 20, 2020

openjdk bot commented Sep 20, 2020

mlbridge bot commented Sep 20, 2020 •

edited

yminqi commented Sep 20, 2020

iklam commented Sep 21, 2020

shipilev Sep 21, 2020

iklam Sep 21, 2020

shipilev Sep 21, 2020

DamonFool commented Sep 21, 2020

DamonFool commented Sep 21, 2020

openjdk bot commented Sep 21, 2020

yminqi left a comment

openjdk bot commented Sep 21, 2020 •

edited

iklam commented Sep 21, 2020

openjdk bot commented Sep 21, 2020

iklam commented Sep 21, 2020

openjdk bot commented Sep 21, 2020

iklam commented Sep 22, 2020

openjdk bot commented Sep 22, 2020

8253079: DeterministicDump.java fails due to garbage in structure padding #267

8253079: DeterministicDump.java fails due to garbage in structure padding #267

Conversation

iklam commented Sep 20, 2020 • edited by openjdk[bot] bot

Progress

Issue

Reviewers

Download

bridgekeeper bot commented Sep 20, 2020

openjdk bot commented Sep 20, 2020

mlbridge bot commented Sep 20, 2020 • edited

Webrevs

yminqi commented Sep 20, 2020

iklam commented Sep 21, 2020

shipilev Sep 21, 2020

Choose a reason for hiding this comment

iklam Sep 21, 2020

Choose a reason for hiding this comment

shipilev Sep 21, 2020

Choose a reason for hiding this comment

DamonFool commented Sep 21, 2020

DamonFool commented Sep 21, 2020

openjdk bot commented Sep 21, 2020

yminqi left a comment

openjdk bot commented Sep 21, 2020 • edited

iklam commented Sep 21, 2020

openjdk bot commented Sep 21, 2020

iklam commented Sep 21, 2020

openjdk bot commented Sep 21, 2020

iklam commented Sep 22, 2020

openjdk bot commented Sep 22, 2020

iklam commented Sep 20, 2020 •

edited by openjdk[bot] bot

mlbridge bot commented Sep 20, 2020 •

edited

openjdk bot commented Sep 21, 2020 •

edited