8275731: CDS archived enums objects are recreated at runtime #6653

iklam · 2021-12-01T20:47:20Z

Background:

In the Java Language, Enums can be tested for equality, so the constants in an Enum type must be unique. Javac compiles an enum declaration like this:

public enum Day {  SUNDAY, MONDAY ... }

to

public class Day extends java.lang.Enum {
    public static final SUNDAY = new Day("SUNDAY");
    public static final MONDAY = new Day("MONDAY"); ...
}

With CDS archived heap objects, Day::<clinit> is executed twice: once during java -Xshare:dump, and once during normal JVM execution. If the archived heap objects references one of the Enum constants created at dump time, we will violate the uniqueness requirements of the Enum constants at runtime. See the test case in the description of JDK-8275731

Fix:

During -Xshare:dump, if we discovered that an Enum constant of type X is archived, we archive all constants of type X. At Runtime, type X will skip the normal execution of X::<clinit>. Instead, we run HeapShared::initialize_enum_klass() to retrieve all the constants of X that were saved at dump time.

This is safe as we know that X::<clinit> has no observable side effect -- it only creates the constants of type X, as well as the synthetic value X::$VALUES, which cannot be observed until X is fully initialized.

Verification:

To avoid future problems, I added a new tool, CDSHeapVerifier, to look for similar problems where the archived heap objects reference a static field that may be recreated at runtime. There are some manual steps involved, but I analyzed the potential problems found by the tool are they are all safe (after the current bug is fixed). See cdsHeapVerifier.cpp for gory details. An example trace of this tool can be found at https://bugs.openjdk.java.net/secure/attachment/97242/enum_warning.txt

Testing:

Passed Oracle CI tiers 1-4. WIll run tier 5 as well.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8275731: CDS archived enums objects are recreated at runtime

Reviewers

Coleen Phillimore (@coleenp - Reviewer) ⚠️ Review applies to c6e9be1
Calvin Cheung (@calvinccheung - Reviewer) ⚠️ Review applies to e27d352

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6653/head:pull/6653
$ git checkout pull/6653

Update a local copy of the PR:
$ git checkout pull/6653
$ git pull https://git.openjdk.java.net/jdk pull/6653/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6653

View PR using the GUI difftool:
$ git pr show -t 6653

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6653.diff

bridgekeeper · 2021-12-01T20:48:39Z

👋 Welcome back iklam! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-12-01T20:50:29Z

@iklam The following label will be automatically applied to this pull request:

hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

iklam · 2021-12-01T20:51:43Z

/label add core-libs

openjdk · 2021-12-01T20:51:52Z

@iklam
The core-libs label was successfully added.

mlbridge · 2021-12-01T21:02:48Z

Webrevs

mlbridge · 2022-01-05T07:40:37Z

Mailing list message from Ioi Lam on hotspot-dev:

Still looking for reviewers ....

Thanks
- Ioi

On 12/1/21 1:02 PM, Ioi Lam wrote:

coleenp

I don't really know this code well enough to do a good code review. I had some comments though.

coleenp · 2022-01-17T19:22:23Z

src/hotspot/share/cds/heapShared.cpp

@@ -377,6 +479,7 @@ void HeapShared::archive_objects(GrowableArray<MemRegion>* closed_regions,
    log_info(cds)("Dumping objects to open archive heap region ...");
    copy_open_objects(open_regions);

+    CDSHeapVerifier::verify();


Should all this be DEBUG_ONLY ?

I changed CDSHeapVerifier::verify() to a NOT_DEBUG_RETURN function.

coleenp · 2022-01-17T19:22:23Z

src/hotspot/share/cds/heapShared.hpp

+    KlassSubGraphInfo* _subgraph_info;
+    oop _referrer;
+    oop _obj;
+    CachedOopInfo() :_subgraph_info(), _referrer(), _obj() {}


Should these be initialized to nullptr? does this do this?

These three fields are initialized with the default initializer (empty parenthesis) so they will be initialized to the null pointer.

coleenp · 2022-01-17T19:22:23Z

src/hotspot/share/cds/cdsHeapVerifier.cpp

+    }
+
+    ResourceMark rm;
+    for (JavaFieldStream fs(ik); !fs.done(); fs.next()) {


Can this call instead
void InstanceKlass::do_local_static_fields(void f(fieldDescriptor*, Handle, TRAPS), Handle mirror, TRAPS) {
and have this next few lines in the function?

I moved the code inside a new class CDSHeapVerifier::CheckStaticFields so I can call InstanceKlass::do_local_static_fields

coleenp · 2022-01-17T19:22:23Z

src/hotspot/share/cds/cdsHeapVerifier.cpp

+    if (k->is_instance_klass()) {
+      InstanceKlass* ik = InstanceKlass::cast(k);
+      for (JavaFieldStream fs(ik); !fs.done(); fs.next()) {
+        if (!fs.access_flags().is_static()) {


same here. It only saves a couple of lines but then you can have the function outside this large function.

You actually found a bug here. I am iterating over non-static fields and should walk the inherited fields as well. I changed the code to call InstanceKlass::do_nonstatic_fields()

coleenp · 2022-01-17T19:22:23Z

src/hotspot/share/cds/cdsHeapVerifier.hpp

+      ResourceObj::C_HEAP,
+      mtClassShared,
+      HeapShared::oop_hash> _table;
+


Is this only used inside cdsHeapVerifier? if so it should be in the .cpp file. There's also an ArchiveableStaticFieldInfo. Not sure how they are related.

This _table is part of the CDSHeapVerifier instance, which is stack allocated. So I need to declare it as part of the CDSHeapVerifier class declaration in the hpp file.

coleenp · 2022-01-17T19:22:23Z

src/hotspot/share/cds/heapShared.cpp

+
+  oop mirror = k->java_mirror();
+  int i = 0;
+  for (JavaFieldStream fs(k); !fs.done(); fs.next()) {


This seems like it should also use InstanceKlass::do_local_static_fields.

Converting this to InstanceKlass::do_nonstatic_fields() is difficult because the loop body references 7 different variables declared outside of the loop.

One thing I tried is to add a new version of do_nonstatic_fields2() that supports C++ lambdas. You can see my experiment from here:

https://github.com/openjdk/jdk/compare/master...iklam:lambda-for-instanceklass-do_local_static_fields2?expand=1

I changed all my new code to use the do_nonstatic_fields2() function with lambda.

Ok, if it requires lambdas and additional change, never mind then.

openjdk · 2022-01-19T05:43:50Z

@iklam this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout 8275731-heapshared-enum
git fetch https://git.openjdk.java.net/jdk master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

iklam · 2022-01-19T05:50:50Z

I don't really know this code well enough to do a good code review. I had some comments though.

Hi Coleen, thanks for taking a look.

This PR has two major parts:

Check for inappropriate reference to static fields. This is mainly done in cdsHeapVerifier.cpp. These checks don't affect the contents of the CDS archive. They just print out warnings if problems are found.
Special initialization of enum classes. Essentially if any instance of an enum class X is archived, then X::<clinit> will not be executed, and we'll take this path instead (in instanceKlass.cpp):

  // This is needed to ensure the consistency of the archived heap objects.
  if (has_archived_enum_objs()) {
    assert(is_shared(), "must be");
    bool initialized = HeapShared::initialize_enum_klass(this, CHECK);
    if (initialized) {
      return;
    }
  }

Could you check if (2) is correct?

bridgekeeper · 2022-02-16T10:36:13Z

@iklam This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

iklam · 2022-02-16T17:45:35Z

@iklam This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

keepalive

calvinccheung · 2022-02-17T23:19:16Z

src/hotspot/share/cds/cdsHeapVerifier.cpp

+//
+// class Bar {
+//     // this field is initialized in both CDS dump time and runtime.
+//     static final Bar bar = new Bar;


new Bar should be new Bar()?

calvinccheung

Looks good. Minor comment below.
Also, several files with copyright year 2021 need updating.

openjdk · 2022-02-23T04:11:24Z

@iklam This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8275731: CDS archived enums objects are recreated at runtime

Reviewed-by: coleenp, ccheung

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been no new commits pushed to the master branch. If another commit should be pushed before you perform the /integrate command, your PR will be automatically rebased. If you prefer to avoid any potential automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

coleenp

Sorry for the long delay. It's a big change, but a lot in debug so that's ok. Looks good.

coleenp · 2022-02-25T14:58:17Z

src/hotspot/share/cds/heapShared.cpp

+
+  oop mirror = k->java_mirror();
+  int i = 0;
+  for (JavaFieldStream fs(k); !fs.done(); fs.next()) {


Ok, if it requires lambdas and additional change, never mind then.

iklam · 2022-02-28T20:32:14Z

Thanks @calvinccheung and @coleenp for the review. Passed tiers 1-5.
/integrate

openjdk · 2022-02-28T20:33:28Z

Going to push as commit d983d10.
Since your change was applied there have been 8 commits pushed to the master branch:

c7cd148: 8282240: Add _name field to Method for NOT_PRODUCT only
59b3ecc: 8277976: Break up SEQUENCE in X509Certificate::getSubjectAlternativeNames and X509Certificate::getIssuerAlternativeNames in otherName
4e7fb41: 8282172: CompileBroker::log_metaspace_failure is called from non-Java/compiler threads
0ae3d1d: 8282131: java.time.ZoneId should be a sealed abstract class
c58f5c6: 8282360: Merge POSIX implementations of ThreadCritical
06cadb3: 8230382: Clean up ConvI2L, CastII and CastLL::Ideal methods
efd3967: 8267265: Use new IR Test Framework to create tests for C2 Ideal transformations
86723d4: 8281507: Two javac tests have bad jtreg @clean tags

Your commit was automatically rebased without conflicts.

openjdk · 2022-02-28T20:33:42Z

@iklam Pushed as commit d983d10.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8275731: CDS archived enums objects are recreated at runtime

c2bb1f2

openjdk bot added the hotspot label Dec 1, 2021

openjdk bot added the core-libs label Dec 1, 2021

iklam marked this pull request as ready for review Dec 1, 2021

openjdk bot added the rfr label Dec 1, 2021

Comments from @calvinccheung off-line

df0d3f8

openjdk bot removed the rfr label Dec 8, 2021

ioilam added 2 commits Dec 10, 2021

added exclusions needed by "java -Xshare:dump -ea -esa"

39c21c6

Merge branch 'master' into 8275731-heapshared-enum

6e16005

coleenp reviewed Jan 17, 2022

View changes

Use InstanceKlass::do_local_static_fields for some field iterations

e27d352

openjdk bot added the merge-conflict label Jan 19, 2022

calvinccheung reviewed Feb 17, 2022

View changes

calvinccheung approved these changes Feb 17, 2022

View changes

ioilam added 2 commits Feb 23, 2022

Merge branch 'master' into 8275731-heapshared-enum

adc6b8a

Fixed comments per @calvinccheung review

4764075

openjdk bot removed the merge-conflict label Feb 23, 2022

fixed whitespace

c6e9be1

openjdk bot added the ready label Feb 23, 2022

openjdk bot added the rfr label Feb 23, 2022

coleenp approved these changes Feb 25, 2022

View changes

ioilam added 2 commits Feb 28, 2022

Merge branch 'master' into 8275731-heapshared-enum

f953225

fixed copyright year

8bc6ddc

openjdk bot added the integrated label Feb 28, 2022

openjdk bot closed this Feb 28, 2022

openjdk bot removed ready rfr labels Feb 28, 2022

openjdk / jdk Public

8275731: CDS archived enums objects are recreated at runtime #6653

8275731: CDS archived enums objects are recreated at runtime #6653

iklam commented Dec 1, 2021 •

edited by openjdk[bot] bot

bridgekeeper bot commented Dec 1, 2021

openjdk bot commented Dec 1, 2021

iklam commented Dec 1, 2021

openjdk bot commented Dec 1, 2021

mlbridge bot commented Dec 1, 2021 •

edited

mlbridge bot commented Jan 5, 2022

coleenp left a comment

coleenp Jan 17, 2022

iklam Jan 19, 2022

coleenp Jan 17, 2022

iklam Jan 19, 2022

coleenp Jan 17, 2022

iklam Jan 19, 2022

coleenp Jan 17, 2022

iklam Jan 19, 2022

coleenp Jan 17, 2022

iklam Jan 19, 2022

coleenp Jan 17, 2022

iklam Jan 19, 2022

coleenp Feb 25, 2022

openjdk bot commented Jan 19, 2022

iklam commented Jan 19, 2022

bridgekeeper bot commented Feb 16, 2022

iklam commented Feb 16, 2022

calvinccheung Feb 17, 2022

calvinccheung left a comment

openjdk bot commented Feb 23, 2022 •

edited

coleenp left a comment

coleenp Feb 25, 2022

iklam commented Feb 28, 2022

openjdk bot commented Feb 28, 2022

openjdk bot commented Feb 28, 2022

openjdk / jdk Public

8275731: CDS archived enums objects are recreated at runtime #6653

8275731: CDS archived enums objects are recreated at runtime #6653

Conversation

iklam commented Dec 1, 2021 • edited by openjdk[bot] bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Dec 1, 2021

openjdk bot commented Dec 1, 2021

iklam commented Dec 1, 2021

openjdk bot commented Dec 1, 2021

mlbridge bot commented Dec 1, 2021 • edited

Webrevs

mlbridge bot commented Jan 5, 2022

coleenp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented Jan 19, 2022

iklam commented Jan 19, 2022

bridgekeeper bot commented Feb 16, 2022

iklam commented Feb 16, 2022

Choose a reason for hiding this comment

calvinccheung left a comment

openjdk bot commented Feb 23, 2022 • edited

coleenp left a comment

Choose a reason for hiding this comment

iklam commented Feb 28, 2022

openjdk bot commented Feb 28, 2022

openjdk bot commented Feb 28, 2022

iklam commented Dec 1, 2021 •

edited by openjdk[bot] bot

mlbridge bot commented Dec 1, 2021 •

edited

openjdk bot commented Feb 23, 2022 •

edited