8261090: Store old classfiles in static CDS archive #3479

calvinccheung · 2021-04-14T00:14:46Z

Please review this RFE for storing old classfiles with major version < 50 in static CDS archive.
During static CDS dump time, old classes won't be verified/rewritten. They will be verified/rewritten during runtime.
Therefore, the _constMethod, _constants, and _cache of old classes must be stored in the RW region of the archive for runtime rewriting. The ConstantPool::remove_unshareable_info will be skipped during dump time and the ConstantPool::restore_unshareable_info will be skipped during runtime for old classes.

Passed tiers 1,2,3,4 tests on mach5.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8261090: Store old classfiles in static CDS archive

Reviewers

Ioi Lam (@iklam - Reviewer) ⚠️ Review applies to a8ddecf
Yumin Qi (@yminqi - Reviewer) ⚠️ Review applies to a8ddecf

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3479/head:pull/3479
$ git checkout pull/3479

Update a local copy of the PR:
$ git checkout pull/3479
$ git pull https://git.openjdk.java.net/jdk pull/3479/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3479

View PR using the GUI difftool:
$ git pr show -t 3479

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3479.diff

calvinccheung · 2021-04-14T00:15:24Z

/label add hotspot-runtime
/label remove hotspot

bridgekeeper · 2021-04-14T00:18:35Z

👋 Welcome back ccheung! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-04-14T00:19:56Z

@calvinccheung
The hotspot-runtime label was successfully added.

openjdk · 2021-04-14T00:21:08Z

@calvinccheung The hotspot label was not set.

mlbridge · 2021-04-14T00:30:02Z

Webrevs

jianglizhou · 2021-04-14T01:05:27Z

src/hotspot/share/memory/metaspaceShared.cpp

@@ -733,7 +756,7 @@ bool MetaspaceShared::try_link_class(Thread* current, InstanceKlass* ik) {
  ExceptionMark em(current);
  Thread* THREAD = current; // For exception macros.
  Arguments::assert_is_dumping_archive();
-  if (ik->is_loaded() && !ik->is_linked() &&
+  if (ik->is_loaded() && !ik->is_linked() && !MetaspaceShared::is_old_class(ik) &&


It would be desirable to support the cases when verification is not enabled, which are not rare. For old classes, bytecode rewriting and class linking can still be done at dump time in those cases, with your current proposed change that includes old classes in the archive image. Otherwise, runtime would have to spend CPU cycles to do rewriting/linking for old classes when not truly necessary.

I think that should be done in a separate RFE. It has security implication -- it may not be a good idea to implement a performance feature that you can get only by disabling verification. This will need more community input and security review.
What we have implemented in this PR is a compromise -- support old classes in CDS as much as we can, without breaking any security mechanisms.
There are also alternatives for further improvement. For example, we can modify the old verifier to store verification constraints in the CDS archive. This will require more work, but will preserve the Java security model.

jianglizhou · 2021-04-14T01:09:55Z

Could you please provide performance numbers? It would be good to understand the performance benefits from archiving the old classes when verification and linking are deferred at runtime. Thanks.

calvinccheung · 2021-04-15T17:10:25Z

I've added some performance numbers in the bug report.
https://bugs.openjdk.java.net/browse/JDK-8261090?focusedCommentId=14413598&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14413598

iklam

Looks good to me. Some suggestions for comments and performance improvement.

iklam · 2021-04-19T17:02:48Z

src/hotspot/share/classfile/verifier.cpp

@@ -284,8 +285,10 @@ bool Verifier::is_eligible_for_verification(InstanceKlass* klass, bool should_ve
    // Can not verify the bytecodes for shared classes because they have
    // already been rewritten to contain constant pool cache indices,
    // which the verifier can't understand.
+    // Bytecodes for shared old classes can be verified because they have
+    // not been rewritten.


Maybe put the above 2 lines under line 290, and change "can be" to "should be"?

// However, bytecodes for shared old classes should be verified because they have // not been rewritten.

iklam · 2021-04-19T17:03:55Z

src/hotspot/share/interpreter/rewriter.cpp

-  if (!DumpSharedSpaces) {
-    assert(!klass->is_shared(), "archive methods must not be rewritten at run time");
+  if (klass->is_shared()) {
+    assert(!klass->is_rewritten(), "rewritten shared classes cannot be rewritten again");


Also add

assert(MetaspaceShared::is_old_class(klass), "only shared old classes aren't rewritten");

iklam · 2021-04-19T17:06:53Z

src/hotspot/share/oops/constantPool.cpp

@@ -138,7 +138,11 @@ void ConstantPool::metaspace_pointers_do(MetaspaceClosure* it) {
  log_trace(cds)("Iter(ConstantPool): %p", this);

  it->push(&_tags, MetaspaceClosure::_writable);
-  it->push(&_cache);
+  if (!pool_holder()->is_rewritten()) {
+    it->push(&_cache, MetaspaceClosure::_writable);


This shouldn't be necessary. All ConstantPoolCache are allocated in RW space by default.

iklam · 2021-04-19T17:11:36Z

src/hotspot/share/oops/klassVtable.cpp

@@ -51,7 +51,7 @@ inline InstanceKlass* klassVtable::ik() const {
 }

 bool klassVtable::is_preinitialized_vtable() {
-  return _klass->is_shared() && !MetaspaceShared::remapped_readwrite();
+  return _klass->is_shared() && !MetaspaceShared::remapped_readwrite() && !MetaspaceShared::is_old_class(ik());


Maybe we can avoid calling MetaspaceShared::is_old_class() by adding an extra bit into klass.hpp:

#if INCLUDE_CDS // Flags of the current shared class. u2 _shared_class_flags; enum { _archived_lambda_proxy_is_available = 2, _has_value_based_class_annotation = 4, + _is_shared_old_klass = 8 }; #endif

I've added the bit and it will be set in MetaspaceShared::linking_required.
Also replaced calling MetaspaceShared::is_old_class() with k->is_shared_old_klass() in several places.

Thanks for your review.

…hareable_info

iklam

LGTM

openjdk · 2021-04-21T16:46:13Z

@calvinccheung This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8261090: Store old classfiles in static CDS archive

Reviewed-by: iklam, minqi

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 73 new commits pushed to the master branch:

9e7c748: 8265700: Regularize throws clauses in BigDecimal
7116321: 8254565: JFR: Incorrect verification of mirror events
f45d460: 8265017: runtime/HiddenClasses/StressHiddenClasses.java timed out on Win* OCI
0136c89: 8265490: Unterminated string passed to FindClass() in hotspot test
325edbc: 8265450: Merge PreservedMarksSet::restore code paths
b337f63: 8037397: RegEx pattern matching loses character class after intersection (&&) operator
07a7510: 8200145: Conditional expression mistakenly treated as standalone
41fc7dd: 8199079: Test javax/swing/UIDefaults/6302464/bug6302464.java is unstable
45c474a: 8168408: Test java/awt/Focus/ActualFocusedWindowTest/ActualFocusedWindowBlockingTest.java fails intermittentently on windows
18ee419: 8198422: Test java/awt/font/StyledMetrics/BoldSpace.java is unstable
... and 63 more: https://git.openjdk.java.net/jdk/compare/ff4997014fe5462dca2b313f3f483400ffee5b62...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

yminqi · 2021-04-21T18:07:48Z

src/hotspot/share/classfile/verifier.cpp

-    !klass->is_shared() &&
+    // However, bytecodes for shared old classes can be verified because
+    // they have not been rewritten.
+    (!(klass->is_shared() && klass->is_rewritten())) &&


the outmost brackets are not necessary. Could your remove it?

Fixed. I also removed the extra include of metaspaceShared.hpp.

yminqi · 2021-04-21T18:21:18Z

test/hotspot/jtreg/runtime/cds/appcds/OldSuperInf.java

+        // create archive with class list
+        OutputAnalyzer output = TestCommon.dump(appJar, appClasses, "-Xlog:class+load,cds=debug,verification=trace");
+        TestCommon.checkExecReturn(output, 0,
+                                   dynamicMode ? true : false,


Why use a ternary operator, is it the variable itself OK here?

I prefer to leave it as is because the third arg to TestCommon.checkExecReturn() is about whether or not to look for the strings in the subsequent args.

Thanks for your review.

yminqi

LGTM, only small nits.

…tag to new tests

calvinccheung · 2021-04-22T15:12:30Z

/integrate

openjdk · 2021-04-22T15:13:55Z

@calvinccheung Since your change was applied there have been 84 commits pushed to the master branch:

159f5e1: 8263636: Add --disable-registry option to jhsdb debugd
6d49cc3: 8265453: SafepointMechanism::should_process() should receive JavaThread*
fa82d47: 8264514: HexFormat implementation tweaks
e16d568: 8265218: trace_method_handle_stub fails to find calling frame on x86
86bd44f: 8257831: Suspend with handshakes
28af31d: 8263668: Update java.time to use instanceof pattern variable
a93d911: 8265607: Avoid decrementing when no Symbol was created in ~SignatureStream
aa29784: 8265332: gtest/LargePageGtests.java OOMEs on -XX:+UseSHM cases
33b6378: 8265101: Remove unnecessary functions in os*.inline.hpp
28c35ae: 8259288: Debug build failure with clang-10 due to -Wimplicit-int-float-conversion
... and 74 more: https://git.openjdk.java.net/jdk/compare/ff4997014fe5462dca2b313f3f483400ffee5b62...master

Your commit was automatically rebased without conflicts.

Pushed as commit 9499175.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

calvinccheung added 3 commits April 13, 2021 21:25

8261090: Store old classfiles in static CDS archive

cbf6921

add comments and some code simplification

7dfad83

Merge branch 'master' into 8261090-store-old-classfiles

f2b4433

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Apr 14, 2021

remove extraneous whitespace

11fdb56

calvinccheung marked this pull request as ready for review April 14, 2021 00:26

openjdk bot added the rfr Pull request is ready for review label Apr 14, 2021

jianglizhou reviewed Apr 14, 2021

View reviewed changes

use is_written() instead of is_old_class() in constantPool.cpp

8c6b61b

iklam suggested changes Apr 19, 2021

View reviewed changes

calvinccheung added 3 commits April 19, 2021 18:15

Merge branch 'master' into 8261090-store-old-classfiles

d897eb1

review comments from @iklam

c93eaed

move the setting of shared old class bit to InstanceKlass::remove_uns…

a8ddecf

…hareable_info

iklam approved these changes Apr 21, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Apr 21, 2021

yminqi reviewed Apr 21, 2021

View reviewed changes

yminqi approved these changes Apr 21, 2021

View reviewed changes

comment from @yminqi, add comment to the is_old_class(), add the @bug …

ea81a6c

…tag to new tests

openjdk bot closed this Apr 22, 2021

openjdk bot added the integrated Pull request has been integrated label Apr 22, 2021

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Apr 22, 2021

calvinccheung deleted the 8261090-store-old-classfiles branch April 22, 2021 15:17

8261090: Store old classfiles in static CDS archive #3479

8261090: Store old classfiles in static CDS archive #3479

Uh oh!

Conversation

calvinccheung commented Apr 14, 2021 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Uh oh!

calvinccheung commented Apr 14, 2021

Uh oh!

bridgekeeper bot commented Apr 14, 2021

Uh oh!

openjdk bot commented Apr 14, 2021

Uh oh!

openjdk bot commented Apr 14, 2021

Uh oh!

mlbridge bot commented Apr 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

jianglizhou Apr 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jianglizhou commented Apr 14, 2021

Uh oh!

calvinccheung commented Apr 15, 2021

Uh oh!

iklam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iklam left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Apr 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yminqi left a comment

Choose a reason for hiding this comment

Uh oh!

calvinccheung commented Apr 22, 2021

Uh oh!

openjdk bot commented Apr 22, 2021

Uh oh!

Uh oh!

calvinccheung commented Apr 14, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Apr 14, 2021 •

edited

Loading

jianglizhou Apr 14, 2021 •

edited

Loading

openjdk bot commented Apr 21, 2021 •

edited

Loading